SYSTEM NOTICE

Auto translation by AI. Be sure, accuracy, nuances and authorial intent may not be fully reflected.
見出し画像

[Home Server Apps] Daily Storage Health Check with Scrutiny

自宅快適化計画

Hello, this is Nobo.
In this post, I will introduce an app called Scrutiny that monitors the health status of HDDs.

The home server I am building in this Note uses TrueNAS as its platform.Ensuring data protection is one of my major goals. To achieve this, I have increased data fault tolerance through software-level protections like ZFS data integrity assurance and snapshot-based data protection, as well as hardware-level redundancy like RAID-1. However, a feature to check the health status of the HDDs themselves is missing.

Up until TrueNAS 25.04, there was a built-in feature to check HDD health using S.M.A.R.T. functions. However, the TrueNAS development team determined that S.M.A.R.T.-based diagnostics were not appropriate, so the feature was removed starting from TrueNAS 25.10. Instead, TrueNAS now recommends using "Scrutiny," which I am introducing today.


About Scrutiny

It is a web application that automates health checks for hard disks and SSDs, detects signs of failure early, and sends alerts.

Official Website: https://github.com/AnalogJ/scrutiny

Main Features:

  1. S.M.A.R.T. Monitoring
    S.M.A.R.T. (Self-Monitoring, Analysis and Reporting Technology) is a technology that allows HDDs and SSDs to monitor their own internal status.

  2. Data Analysis
     It compares the acquired S.M.A.R.T. information with Backblaze's large-scale failure data statistics to evaluate actual failure risk.

  3. Beautiful Web Dashboard and Notification Features
    It displays the above information in an easy-to-understand web interface and notifies you of necessary details using your preferred method.

Web Dashboard:

The main dashboard looks like this and is easy to understand.
(Although one of the HDDs is marked as "Failed"...)
The top half shows the status of each storage device, and the bottom half displays the temperature trends of the HDDs/SSDs as a graph.

You can check the storage status at a glance

Here are the details of the HDD marked as Failed. The S.M.A.R.T. information shows an abnormal number of "Command Timeout" occurrences, but the HDD is functioning without any issues.

Abnormal value for Command Timeout

Installation and Configuration

It is very simple, so I will only explain the key points.

Installation on TrueNAS

  • Install "Scrutiny" from the TrueNAS App Market

    • Add the following to Additional Environment Variables

      • Name: COLLECTOR_CRON_SCHEDULE

      • Value: 0 * * * * (Set the notification timing)

    • Everything else can be left as default

Scrutiny Configuration

  • Creating configuration files
    Refer to the Scrutiny GitHub and create the following two files.
    Basically, you just need to copy the provided content, but you will need to edit the notification section. (It would be nice if it were a GUI.)

    • scrutiny.yaml
      This is the file for main settings. Here, you set the "notification method."

      • notify: -> # Uncomment

      • urls: -> # Uncomment

      • Set your preferred notification method -> # Uncomment and configure
        The configuration details are supplemented below.

    • collector.yaml
      This is used for detailed settings of target devices or when managing multiple hosts together (hub-and-spoke configuration). I have not specifically configured this for now. Please edit as needed.

Supplement: Scrutiny provides a variety of notification methods.
It is also possible to execute scripts.

  • Example of notification settings using scrutiny.yaml

    • Example 1:Notification to Discord
      Create a Webhook on your own Discord server.
      Set the "token" and "webhookid" from the Webhook URL.

    • Example 2:Notification to Slack
      Create a "Bot User OAuth Token" from your Slack App, and set the "Bot Name" and "Token".

    • Example 3:SMTP notification (Email)
      For Gmail, create an App Password, and set the "Mail address", "Password", "smtp.gmail.com", "587", "Sender", and "Recipient".

#notify:
#  urls:
#    - "discord://token@webhookid"
#    - "telegram://token@telegram?channels=channel-1[,channel-2,...]"
#    - "pushover://shoutrrr:apiToken@userKey/?priority=1&devices=device1[,device2, ...]"
#    - "slack://[botname@]token-a/token-b/token-c"
#    - "smtp://username:password@host:port/?fromAddress=fromAddress&toAddresses=recipient1[,recipient2,...]"
#    - "teams://token-a/token-b/token-c"
#    - "gotify://gotify-host/token"
#    - "pushbullet://api-token[/device/#channel/email]"
#    - "ifttt://key/?events=event1[,event2,...]&value1=value1&value2=value2&value3=value3"
#    - "mattermost://[username@]mattermost-host/token[/channel]"
#    - "ntfy://username:password@host:port/topic"
#    - "hangouts://chat.googleapis.com/v1/spaces/FOO/messages?key=bar&token=baz"
#    - "zulip://bot-mail:bot-key@zulip-domain/?stream=name-or-id&topic=name"
#    - "join://shoutrrr:api-key@join/?devices=device1[,device2, ...][&icon=icon][&title=title]"
#    - "script:///file/path/on/disk"
#    - "https://www.example.com/path"

  • Add the configuration file to the mount point
    Check the Scrutiny mount point on the TrueNAS app list screen.
    Since the configuration file should be placed in "/opt/scrutiny/config", place the two configuration files created above into "/mnt/App/config/scrutiny", which is bound to that location.
    Honestly, it is easiest to just create them directly using nano and paste the content.

Click the folder-shaped icon
Check the directory bound to "/opt/scrutiny/config"

  • Restarting Scrutiny should apply the settings.

Notification test

Check if the settings are correctly applied and if notifications are received.
If you receive a notification using the method you set up, you are good to go.

  • How to call the Scrutiny API from the command line.

curl -X POST http://（ホスト）:（ポート）/api/health/notify

  • You can also enter the Docker container and execute it directly with the following command.

/opt/scrutiny/bin/scrutiny-collector-metrics run

2026/xx/xx xx:xx:xx Loading configuration file: /opt/scrutiny/config/collector.yaml

 ___   ___  ____  __  __  ____  ____  _  _  _  _
/ __) / __)(  _ \(  )(  )(_  _)(_  _)( \( )( \/ )
\__ \( (__  )   / )(__)(   )(   _)(_  )  (  \  /
(___/ \___)(_)\_)(______) (__) (____)(_)\_) (__)
AnalogJ/scrutiny/metrics                                dev-0.8.6

INFO[0000] Verifying required tools                      type=metrics
INFO[0000] Executing command: smartctl --scan --json     type=metrics
INFO[0000] Executing command: smartctl --info --json /dev/sda  type=metrics
INFO[0000] Generating WWN                                type=metrics
INFO[0000] Executing command: smartctl --info --json /dev/sdb  type=metrics
INFO[0000] Generating WWN                                type=metrics
INFO[0000] Executing command: smartctl --info --json /dev/sdc  type=metrics
INFO[0000] Generating WWN                                type=metrics
INFO[0000] Executing command: smartctl --info --json /dev/sdd  type=metrics
INFO[0000] Generating WWN                                type=metrics
INFO[0000] Executing command: smartctl --info --json --device nvme /dev/nvme0  type=metrics
INFO[0000] Using WWN Fallback                            type=metrics
INFO[0000] Executing command: smartctl --info --json --device nvme /dev/nvme1  type=metrics
INFO[0000] Using WWN Fallback                            type=metrics
INFO[0000] Sending detected devices to API, for filtering & validation  type=metrics
INFO[0000] Collecting smartctl results for sda           type=metrics
INFO[0000] Executing command: smartctl --xall --json --device sat /dev/sda  type=metrics
INFO[0000] Publishing smartctl results for 0x5000c500e37aaf69  type=metrics
INFO[0001] Collecting smartctl results for sdb           type=metrics
INFO[0001] Executing command: smartctl --xall --json --device sat /dev/sdb  type=metrics
INFO[0002] Publishing smartctl results for 0x50024e920179ce90  type=metrics
INFO[0002] Collecting smartctl results for sdc           type=metrics
INFO[0002] Executing command: smartctl --xall --json --device sat /dev/sdc  type=metrics
INFO[0002] Publishing smartctl results for 0x5000c500e9f34108  type=metrics
INFO[0002] Collecting smartctl results for sdd           type=metrics
INFO[0002] Executing command: smartctl --xall --json --device sat /dev/sdd  type=metrics
INFO[0002] Publishing smartctl results for 0x5000c500e9f9fe6f  type=metrics
INFO[0003] Collecting smartctl results for nvme0         type=metrics
INFO[0003] Executing command: smartctl --xall --json --device nvme /dev/nvme0  type=metrics
INFO[0003] Publishing smartctl results for 2444cw400178  type=metrics
INFO[0003] Collecting smartctl results for nvme1         type=metrics
INFO[0003] Executing command: smartctl --xall --json --device nvme /dev/nvme1  type=metrics
INFO[0003] Publishing smartctl results for 2444cw402050  type=metrics
INFO[0003] Main: Completed                               type=metrics

With this, you will receive notifications if any abnormalities occur.

Is Scrutiny necessary?

It might be counter-intuitive to say this after writing all this, but I think it might not be necessary to use it.

The purpose of introducing Scrutiny is to "predict sudden storage death." I am not denying the use of Backblaze's statistical data in addition to S.M.A.R.T. information for that purpose, but the issue is how reliable S.M.A.R.T. information is in the first place.

Reliability of S.M.A.R.T.

One of my HDDs is marked as "Failed" when using Scrutiny and needs to be replaced. However, I know the background reason for this. For a while, I had it in an old NAS and left it for a long time without proper setup. During that time, "Command Timeout" situations accumulated, resulting in abnormal data, but the HDD works perfectly fine for Read/Write operations. In this case, the S.M.A.R.T. information becomes noise.

Additionally, two SSDs that have been running constantly for over half a year show a power-on time of only 58 days, so I am skeptical about whether S.M.A.R.T. is information worth trusting.

S.M.A.R.T. reset issue

Recently, refurbished HDDs have been appearing on the market. Perhaps because HDD durability has improved, there are cases where HDDs previously used for other purposes are sold after having their S.M.A.R.T. information reset. If that is the case, isn't it impossible to judge whether the S.M.A.R.T. information is correct in the first place? (I would like to reset the Command Timeout mentioned above, though.)

Comprehensive data protection by TrueNAS is sufficient

One of the reasons for using TrueNAS is that it thoroughly checks data integrity. This is a ZFS feature, so it is not limited to TrueNAS, but it detects all data errors caused by other I/O errors even if the HDD itself has no problems, ensuring that the written data can be read correctly.
Other than that, I think the TrueNAS platform, which provides features for comprehensive data protection including periodic HDD scrubbing, snapshots, and RAID functions, is sufficient.

The significance of Scrutiny

Of course, I don't think checking S.M.A.R.T. information is meaningless. If the information provided is in the correct state (not reset), I think it has a certain significance, but it is ultimately just reference information and is nothing more than 'nice to have' is my understanding.

Rather than that, I think it is a much more realistic measure to practice the 3-2-1 backup rule of taking multiple backups on different media and, if possible, keeping them in a remote location. Storage devices will definitely break someday. The point is that measures to ensure you don't lose important data based on that premise are what's important.

Although it is discussed on the TrueNAS forums and elsewhere, managing health status with S.M.A.R.T. has been the standard for file servers and the like until now, so Scrutiny is an app for those who feel uneasy without S.M.A.R.T.

I don't think there's any harm in installing it, but there's no problem if you don't.
TrueNAS is excellent, so if there is an error, TrueNAS will notice it and notify you. What is important is an appropriate backup system and an excellent data management platform and with that, I would like to conclude this topic for now.

Data errors on the Minisforum N5 Pro

As an aside, information has been confusing since the end of last year regarding the fact that HDD data errors occur on the Pro version of the Minisforum N5 that I recommend in this Note. Originally, those errors were discovered due to ZFS features, and it is thought that they would not have been noticed with S.M.A.R.T. or other formats like ext4.

Minisforum N5 Pro data errors, but in reality, it has become clear that it is not an HDD error, but a DMA malfunction under specific conditions.
For a time, the SATA controller JMB585 was considered the cause, but that seems to be a false accusation. Errors do not occur on the Minisforum N5 (non-Pro) which uses the same chip. The difference between the non-Pro and Pro is just the CPU, ECC memory support, and motherboard configuration such as PCIe lanes, so I think that is the cause.

At present, it seems that the problem does not occur if you bypass DMA in the kernel's IOMMU settings. (Or should I say it just can't be detected?) Since I have configured IOMMU for GPU passthrough in this 'Supreme Home Server', I am concerned about whether the errors are just not being detected because of that relationship.
Since the root cause is unknown, I would like the truth to be clarified soon.

いいなと思ったら応援しよう！