Error Recovery Level

The Linux SCSI Target Wiki

The Error Recovery Level (ERL) is negotiated during a leading iSCSI connection login in traditional iSCSI (RFC 3720) and iSER (RFC 5046).

Contents 1 Hierarchy 2 Implementation 2.1 ERL=0: Session Recovery 2.1.1 Session recovery 2.1.2 Session reinstatement 2.2 ERL=1: Digest Failure Recovery 2.2.1 Within Connection Recovery 2.2.2 Within Command Recovery 2.2.3 Logic to handle recovery / generate R2Ts 2.2.4 Logic to issue SNACK for missing StatSN or DataIN 2.3 ERL=2: Connection Recovery 2.3.1 Connection recovery 2.3.2 Session reinstatement 3 RFCs 4 See also 5 Notes 6 External links

Hierarchy

The following diagram illustrates the RFC 3270 "Error Recovery Hierarchy":^[1]

                         +
                        / \
                       / 2 \       <-- Connection recovery
                      +-----+
                     /   1   \     <-- Digest failure recovery
                    +---------+
                   /     0     \   <-- Session failure recovery
                  +-------------+

The following table provides an overview over the error recovery capabilities expected from each error recovery level implementation respectively.

ERL	Associated Error Recovery Capabilities
0	Session recovery class (Section 6.1.4.4 - Session Recovery)
1	Digest failure recovery (see note below) plus ERL=0
2	Connection recovery class (Section 6.1.4.3 - Connection Recovery) plus ERL=1

Note: Digest failure recovery is comprised of two recovery classes: Within-Connection recovery class (Section 6.1.4.2 Recovery Within-connection) and Within-Command recovery class (RFC 3270 Section 6.1.4.1 "Recovery Within-command").

Implementation

The following sections provide more detail over the error recovery capabilities expected from each error recovery level implementation.

ERL=0: Session Recovery

ERL=0 (Session Recovery, iscsi_target_erl0.c) is triggered when failures within a command, within a connection, and/or within TCP occur. This causes all of the previous connections from the failed session to be restarted on a new session by sending a iSCSI Login Request with a zero TSIH.

Session recovery

Restart all iSCSI connections on any failure.

Session reinstatement

This is a special case for ERL=0 and recovering the existing I_T Nexus.

ERL=1: Digest Failure Recovery

ERL=1 (Digest Failure Recovery, iscsi_target_erl1.c) only applies to traditional iSCSI. For iSCSI/SCTP (which has its own CRC32C) and both types of iSER (so far), handling header and data checksum recovery can be disabled.

Within Connection Recovery

• CmdSN Retry Timer
• Logic to handle Recovery R2T

Within Command Recovery

• DataOut Timer
• Datain Timer

Logic to handle recovery / generate R2Ts

Logic to issue SNACK for missing StatSN or DataIN

• Support RDATA SNACK [this is still TODO]

ERL=2: Connection Recovery

ERL=2 (Connection Recovery, iscsi_target_erl2.c), also known as "Internexus-MP", is an optional RFC-3720 feature that allows for both single and multiple communication path sessions within a iSCSI I_T Nexus (and hence the SCSI Nexus) to actively perform realligence/retry on iSCSI ITTs from failed iSCSI connections. ERL=2 allows iSCSI fabrics to take advantage of recovery in all regards of transport level fabric failures, and in a completely OS independent fashion (i.e. below the host OS storage stack).

As being a SCSI feature, it is generic to the underlying network protocol (fabric module), and has been implemented with iSCSI/TCP, iSCSI/SCTP, and is possible for iSER/DDP, iSER/IB. With the latter case (iSER) traditional iSCSI recovery logic that pertains to ERL=1 is disabled, as the underlying RCaP is handling integrity of payloads using CRC32C or greater checking.

Note that in ERL=0 sessions, all communication paths need to be shutdown/restarted after a recovery exception occurs.

Connection recovery

Handle Logout Request

REMOVECONNFORRECOVERY (CSM-E)

Handle generation of Recovery R2Ts for WRITE

Traditional iSCSI, iSER

Handle recovery DATAIN for READ

Traditional iSCSI, iSER

Handle changed MaxRecvDataSegmentLength across ERL=2

Traditional iSCSI [TODO], iSER [TODO]

Session reinstatement

Handle new Login Request for existing iSCSI connection handle (CSM-I).

RFCs

• RFC 3720: Internet Small Computer Systems Interface (iSCSI)
• RFC 5046: Internet Small Computer Systems Interface (iSCSI) Extensions for Remote Direct Memory Access (RDMA)

See also

• Target
• SCSI: Persistent Reservations (PR), Asymmetric Logical Unit Assignment (ALUA), Multiple Connections per Session (MC/S)
• Fabric modules: Fibre Channel, FCoE, iSCSI, InfiniBand and vHost
• Management: targetcli

Notes

External links

• RFC 3720: Internet Small Computer Systems Interface (iSCSI)
• RFC 5046: Internet Small Computer System Interface (iSCSI) Extensions for Remote Direct Memory Access (RDMA)