Table of Contents

Foreword by Christos Karamanolis xvii

About the Author xix

About the Technical Reviewers xxi

Acknowledgments xxiii

We Want to Hear from You! xxv

1 Introduction to VSAN 1

Software-Defined Datacenter 1

Software-Defined Storage 2

Hyper-Convergence/Server SAN Solutions 3

Introducing Virtual SAN 4

What Is Virtual SAN? 6

What Does VSAN Look Like to an Administrator? 9

Summary 12

2 VSAN Prerequisites and Requirements for Deployment 13

VMware vSphere 13

ESXi 14

Cache and Capacity Devices 14

ESXi Boot Considerations 15

VSAN Requirements 15

VMware Hardware Compatibility Guide 16

VSAN Ready Nodes 16

Storage Controllers 17

Capacity Tier Devices 19

Cache Tier Devices 21

Network Requirements 22

Network Interface Cards 22

Supported Virtual Switch Types 22

Layer 2 or Layer 3 23

VMkernel Network 23

VSAN Network Traffic 24

Jumbo Frames 24

NIC Teaming 25

Network I/O Control 25

VSAN Stretched Cluster 25

VSAN 2-Node Remote Office/Branch Office (ROBO) 26

Firewall Ports 26

Summary 26

3 VSAN Installation and Configuration 29

VSAN Networking 29

VMkernel Network for VSAN 30

VSAN Network Configuration: VMware Standard Switch 31

VSAN Network Configuration: vSphere Distributed Switch 32

Step 1: Create the Distributed Switch 32

Step 2: Create a Distributed Port Group 33

Step 3: Build VMkernel Ports 34

Possible Network Configuration Issues 38

Network I/O Control Configuration Example 40

Design Considerations: Distributed Switch and Network I/O Control 42

Scenario 1: Redundant 10 GbE Switch Without "Link Aggregation" Capability 43

Scenario 2: Redundant 10 GbE Switch with Link Aggregation Capability 45

Creating a VSAN Cluster 48

vSphere HA 49

vSphere HA Communication Network 49

vSphere HA Heartbeat Datastores 50

vSphere HA Admission Control 50

vSphere HA Isolation Response 51

vSphere HA Component Protection 51

The Role of Disk Groups 51

Disk Group Maximums 52

Why Configure Multiple Disk Groups in VSAN? 52

Cache Device to Capacity Device Sizing Ratio 53

Automatically Add Disks to VSAN Disk Groups 54

Manually Adding Disks to a VSAN Disk Group 55

Disk Group Creation Example 55

VSAN Datastore Properties 58

Summary 59

4 VM Storage Policies on VSAN 61

Introducing Storage Policy-Based Management in a VSAN Environment 62

Number of Failures to Tolerate 65

Failure Tolerance Method 66

Number of Disk Stripes Per Object 69

IOPS Limit for Object 70

Flash Read Cache Reservation 71

Object Space Reservation 71

Force Provisioning 71

Disable Object Checksum 73

VASA Vendor Provider 73

An Introduction to VASA 73

Storage Providers 74

VSAN Storage Providers: Highly Available 75

Changing VM Storage Policy On-the-Fly 75

Objects, Components, and Witnesses 80

VM Storage Policies 80

Enabling VM Storage Policies 81

Creating VM Storage Policies 81

Assigning a VM Storage Policy During VM Provisioning 81

Summary 82

5 Architectural Details 83

Distributed RAID 83

Objects and Components 86

Component Limits 87

Virtual Machine Storage Objects 88

Namespace 89

Virtual Machine Swap 90

VMDKs and Deltas 90

Witnesses and Replicas 90

Object Layout 91

VSAN Software Components 94

Component Management 95

Data Paths for Objects 95

Object Ownership 96

Placement and Migration for Objects 96

Cluster Monitoring, Membership, and Directory Services 97

Host Roles (Master, Slave, Agent) 97

Reliable Datagram Transport 98

On-Disk Formats 98

Cache Devices 99

Capacity Devices 99

VSAN I/O Flow 100

Caching Algorithms 100

The Role of the Cache Layer 100

Anatomy of a VSAN Read on Hybrid VSAN 102

Anatomy of a VSAN Read on All-Flash VSAN 103

Anatomy of a VSAN Write on Hybrid VSAN 103

Anatomy of a VSAN Write on All-Flash VSAN 104

Retiring Writes to Capacity Tier on Hybrid VSAN 105

Deduplication and Compression 105

Data Locality 107

Data Locality in VSAN Stretched Clusters 108

Storage Policy-Based Management 109

VSAN Capabilities 109

Number of Failures to Tolerate Policy Setting 110

Best Practice for Number of Failures to Tolerate 112

Stripe Width Policy Setting 113

RAID-0 Used When No Striping Specified in the Policy 117

Stripe Width Maximum 119

Stripe Width Configuration Error 120

Stripe Width Chunk Size 121

Stripe Width Best Practice 122

Flash Read Cache Reservation Policy Setting 122

Object Space Reservation Policy Setting 122

VM Home Namespace Revisited 123

VM Swap Revisited 123

How to Examine the VM Swap Storage Object 124

Delta Disk / Snapshot Caveat 126

Verifying How Much Space Is Actually Consumed 126

Force Provisioning Policy Setting 127

Witnesses and Replicas: Failure Scenarios 127

Data Integrity Through Checksum 130

Recovery from Failure 131

Problematic Device Handling 134

What About Stretching VSAN? 134

Summary 135

6 VM Storage Policies and Virtual Machine Provisioning 137

Policy Setting: Number of Failures to Tolerate = 1 137

Policy Setting: Failures to Tolerate = 1, Stripe Width = 2 144

Policy Setting: Failures to Tolerate = 2, Stripe Width = 2 148

Policy Setting: Failures to Tolerate = 1, Object Space Reservation = 50% 152

Policy Setting: Failures to Tolerate = 1, Object Space Reservation = 100% 155

Policy Setting: RAID-5 157

Policy Setting: RAID-6 158

Policy Setting: RAID-5/6 and Stripe Width = 2 159

Default Policy 160

Summary 164

7 Management and Maintenance 165

Health Check 165

Health Check Tests 165

Proactive Health Checks 167

Performance Service 168

Host Management 169

Adding Hosts to the Cluster 169

Removing Hosts from the Cluster 170

ESXCLI VSAN Cluster Commands 171

Maintenance Mode 172

Default Maintenance Mode/Decommission Mode 175

Recommended Maintenance Mode Option for Updates and Patching 175

Disk Management 177

Adding a Disk Group 177

Removing a Disk Group 178

Adding Disks to the Disk Group 179

Removing Disks from the Disk Group 180

Wiping a Disk 182

Blinking the LED on a Disk 183

ESXCLI VSAN Disk Commands 184

Failure Scenarios 185

Capacity Device Failure 185

Cache Device Failure 186

Host Failure 187

Network Partition 188

Disk Full Scenario 193

Thin Provisioning Considerations 194

vCenter Management 195

vCenter Server Failure Scenario 196

Running vCenter Server on VSAN 196

Bootstrapping vCenter Server 197

Summary 199

8 Stretched Cluster 201

What is a Stretched Cluster? 201

Requirements and Constraints 203

Networking and Latency Requirements 205

New Concepts in VSAN Stretched Cluster 206

Configuration of a Stretched Cluster 208

Failure Scenarios 216

Summary 224

9 Designing a VSAN Cluster 225

Ready Node Profiles 225

Sizing Constraints 227

Cache to Capacity Ratio 228

Designing for Performance 229

Impact of the Disk Controller 231

VSAN Performance Capabilities 235

Design and Sizing Tools 236

Scenario 1: Server Virtualization-Hybrid 237

Determining Your Host Configuration 238

Scenario 2-Server Virtualization-All-flash 241

Summary 244

10 Troubleshooting, Monitoring, and Performance 245

Health Check 246

Ask VMware 246

Health Check Categories 247

Proactive Health Checks 253

ESXCLI 256

esxcli vsan datastore 256

esxcli vsan network 257

esxcli vsan storage 258

esxcli vsan cluster 262

esxcli vsan faultdomain 263

esxcli vsan maintenancemode 264

esxcli vsan policy 264

esxcli vsan trace 267

Additional Non-ESXCLI Commands for Troubleshooting VSAN 268

Ruby vSphere Console 275

VSAN Commands 276

SPBM Commands 300

Troubleshooting VSAN on the ESXi 303

Log Files 304

VSAN Traces 304

VSAN VMkernel Modules and Drivers 305

Performance Monitoring 305

Introducing the Performance Service 305

ESXTOP Performance Counters for VSAN 308

vSphere Web Client Performance Counters for VSAN 309

VSAN Observer 310

Sample VSAN Observer Use Case 316

Summary 318

TOC, 9780134511665, 5/10/2016

About the Author

Cormac Hogan is a Senior Staff Engineer in the Office of the CTO in the Storage and Availability business unit at VMware. Cormac was one of the first VMware employees at the EMEA headquarters in Cork, Ireland, back in 2005, and has previously held roles in VMware's Technical Marketing, Integration Engineering and Support organizations. Cormac has written a number of storage-related white papers and has given numerous presentations on storage best practices and new features. Cormac is the owner of CormacHogan.com, a blog site dedicated to storage and virtualization. He can be followed on twitter @CormacJHogan. Duncan Epping is a Chief Technologist working for VMware in the Office of CTO of the Storage and Availability business unit. Duncan is responsible for ensuring VMware's future innovations align with essential customer needs, translating customer problems to opportunities, and function as the global lead evangelist for Storage and Availability. Duncan specializes in Software Defined Storage, hyper-converged infrastructures and business continuity/disaster recovery solutions. He has four patents pending and one granted on the topic of availability, storage and resource management. Duncan is the author/owner of VMware Virtualization blog Yellow-Bricks.com and has various books on the topic of VMware including the "vSphere Clustering Deepdive" series. He can be followed on twitter @DuncanYB.