OSb: OSv on BitVisor

OSb: OSv on BitVisor@ysmoo

Saturday, October 18, 14

BitVisor

BitVisor

Hardware

OS

App App App

http://www.justis.as-1.co.jp

Saturday, October 18, 14

OSv

VMM

Hardware

OSvApp

http://medical-care.feed.jp

Saturday, October 18, 14

BitVisor

Saturday, October 18, 14

BitVisor OSv

Saturday, October 18, 14

BitVisor OSv

http://www.root.ne.jp/nishide/shs/

Saturday, October 18, 14

OSv on BitVisor

BitVisor

Hardware

OSvApp

Saturday, October 18, 14

OSv on BitVisor

BitVisor

Hardware

OSvApp

OSbSaturday, October 18, 14

OSv on BitVisor

BitVisor

Hardware

OSvApp

OSb

What the fuckno Virtio !

Physical interfacesucks !

What the hell isPRO/1000 !

Saturday, October 18, 14

OSv on BitVisor

BitVisor

Hardware

OSvApp

OSb

What the fuckno Virtio !

Physical interfacesucks !

What the hell isPRO/1000 !

You have no choice.

Virtio

Saturday, October 18, 14

~ The road to OSb ~

Saturday, October 18, 14

• Boot process

• Some drivers

OSv code reading

Saturday, October 18, 14

• Boot process

• Some drivers

OSv code reading- MBR- Read local disk with INT13/42.- Load command line, boot loader, OSv kernel.- Get memory map with INT15/E820.- Setup segment descriptors/page tables.- Switch to 64-bit mode.- premain()- main()...

Saturday, October 18, 14

• Boot process

• Some drivers

OSv code reading- MBR- Read local disk with INT13/42.- Load command line, boot loader, OSv kernel.- Get memory map with INT15/E820.- Setup segment descriptors/page tables.- Switch to 64-bit mode.- premain()- main()...

- Serial / VGA output.- SATA.- Virtio NIC/BLK/RNG/SCSI.- ACPI.- APIC....

Saturday, October 18, 14

• Boot process

• Some drivers

OSv code reading- MBR- Read local disk with INT13/42.- Load command line, boot loader, OSv kernel.- Get memory map with INT15/E820.- Setup segment descriptors/page tables.- Switch to 64-bit mode.- premain()- main()...

- Serial / VGA output.- SATA.- Virtio NIC/BLK/RNG/SCSI.- ACPI.- APIC....

Straightforward...

Saturday, October 18, 14

Booting OSb

./scripts/run.py

Saturday, October 18, 14

Booting OSb

./scripts/run.py

VMM

OSv

In the case of normal OSv...

Saturday, October 18, 14

Booting OSb

Hardware

./scripts/run.py

Saturday, October 18, 14

Booting OSb

Hardware

./scripts/run.py

ImageUpdate

Saturday, October 18, 14

Booting OSb

Hardware

./scripts/run.py

ImageUpdate

IPMI

Saturday, October 18, 14

Booting OSb

Hardware

./scripts/run.py

ImageUpdate

IPMIPXE Boot

Saturday, October 18, 14

Booting OSb

Hardware

./scripts/run.py

ImageUpdate

IPMIPXE Boot

BitVisor

Saturday, October 18, 14

Booting OSb

Hardware

./scripts/run.py

ImageUpdate

IPMIPXE Boot

BitVisor

OSvNetwork Boot

Saturday, October 18, 14

Booting OSb

Hardware

./scripts/run.py

ImageUpdate

IPMIPXE Boot

BitVisor

OSvNetwork Boot

App

Saturday, October 18, 14

Booting OSb

BitVisor

Hardware

OSvApp

Network Boot

IPMI

./scripts/run.py

TTY

ImageUpdate

PXE Boot

Saturday, October 18, 14

ACPI workaround

drivers/acpi.cc:

// Copy the root table list to dynamic memory if (!is_bitvisor()) { status = AcpiReallocateRootTable(); if (ACPI_FAILURE(status)) { acpi_e("AcpiReallocateRootTable failed: %s\n", AcpiFormatException(status)); return; } }

Saturday, October 18, 14

ACPI workaround

drivers/acpi.cc:

// Copy the root table list to dynamic memory if (!is_bitvisor()) { status = AcpiReallocateRootTable(); if (ACPI_FAILURE(status)) { acpi_e("AcpiReallocateRootTable failed: %s\n", AcpiFormatException(status)); return; } }

Triple fault

Saturday, October 18, 14

ACPI workaround

drivers/acpi.cc:

// Copy the root table list to dynamic memory if (!is_bitvisor()) { status = AcpiReallocateRootTable(); if (ACPI_FAILURE(status)) { acpi_e("AcpiReallocateRootTable failed: %s\n", AcpiFormatException(status)); return; } }

Triple faultSkip in case of OSb

Saturday, October 18, 14

ACPI workaround

drivers/acpi.cc:

// Copy the root table list to dynamic memory if (!is_bitvisor()) { status = AcpiReallocateRootTable(); if (ACPI_FAILURE(status)) { acpi_e("AcpiReallocateRootTable failed: %s\n", AcpiFormatException(status)); return; } }

Triple faultSkip in case of OSb

Machine-specific...?

Saturday, October 18, 14

Hello world !

Saturday, October 18, 14

Performance?

20.75& 21.20&

0.00&

5.00&

10.00&

15.00&

20.00&

25.00&

OSb& OSv&

Elap

sed(Time((sec)�

for((12(billion(6mes)(val++;�

Saturday, October 18, 14

Virtio NIC

BitVisor

Hardware

OSvApp Actually PRO/1000...

Saturday, October 18, 14

Virtio NIC

BitVisor

Hardware

OSvApp

Virtio NIC

Actually PRO/1000...

Fake Virtio NIC...

Saturday, October 18, 14

Virtio NIC

BitVisor

Hardware

OSvApp

Virtio NIC

Actually PRO/1000...

Fake Virtio NIC...

Hook PCI config access, MMIO, PIO and fake!Saturday, October 18, 14

FakingPCI configuration space

• Faking IDs

• Faking BARs

• Faking capabilities

Saturday, October 18, 14

Faking IDs 0x1AF4(Virtio Device)

Saturday, October 18, 14

Faking IDs 0x1AF4(Virtio Device)

0x1000(Virtio NIC)

Saturday, October 18, 14

Faking IDs 0x1AF4(Virtio Device)

0x1000(Virtio NIC)

0 (Legacy Virtio)

Saturday, October 18, 14

Faking IDs 0x1AF4(Virtio Device)

0x1000(Virtio NIC)

0 (Legacy Virtio)

1 (Virtio NIC)

Saturday, October 18, 14

Faking BARsBAR0: IO spacefor Virtio

BAR1: MMIO spacefor MSI-X

Saturday, October 18, 14

Faking BARsBAR0: IO spacefor Virtio

BAR1: MMIO spacefor MSI-X

= PRO/1000 IO space

Saturday, October 18, 14

Faking BARsBAR0: IO spacefor Virtio

BAR1: MMIO spacefor MSI-X

= PRO/1000 IO space

= PRO/1000 MMIO space

Saturday, October 18, 14

Faking capabilities

Add MSI-Xcapability

Saturday, October 18, 14

Faking capabilities

Add MSI-Xcapability Support

3 vectors

Saturday, October 18, 14

Faking capabilities

Add MSI-Xcapability Support

3 vectors

Vector 0:Reception

Saturday, October 18, 14

Faking capabilities

Add MSI-Xcapability Support

3 vectors

Vector 0:Reception

Vector 1:Transmission

Saturday, October 18, 14

Faking capabilities

Add MSI-Xcapability Support

3 vectors

Vector 0:Reception

Vector 1:Transmission

Vector 2:Control

Saturday, October 18, 14

Virtio NIC Operations

• Virtio ring

• Packet transmission

• Packet reception

Saturday, October 18, 14

Virtio ringControlled bythese registers

BAR0(IO)

Saturday, October 18, 14

Virtio ringControlled bythese registers

3 ringsavailable

BAR0(IO)

Saturday, October 18, 14

Virtio ringControlled bythese registers

3 ringsavailable

Ring 0: ReceptionBAR0(IO)

Saturday, October 18, 14

Virtio ringControlled bythese registers

3 ringsavailable

Ring 0: Reception

Ring 1: Transmission

BAR0(IO)

Saturday, October 18, 14

Virtio ringControlled bythese registers

3 ringsavailable

Ring 0: Reception

Ring 1: Transmission

Ring 2: Control

BAR0(IO)

Saturday, October 18, 14

Virtio ringControlled bythese registers

3 ringsavailable

Ring 0: Reception

Ring 1: Transmission

Ring 2: Control

BAR0(IO)

(Memory)

Saturday, October 18, 14

Virtio ringControlled bythese registers

3 ringsavailable

Ring 0: Reception

Ring 1: Transmission

Ring 2: Control

BAR0(IO)

(Memory)Driver 2 Device

Device 2 Driver

Saturday, October 18, 14

Packet transmission

OSv(Driver)

BitVisor(Device)

PRO/1000

Saturday, October 18, 14

Packet transmission

OSv(Driver)

BitVisor(Device)

Req.

PRO/1000

Req.Req.

Saturday, October 18, 14

Packet transmission

OSv(Driver)

BitVisor(Device)

Req.

PRO/1000

Req.Req.

Req.Req.Req.

Saturday, October 18, 14

Packet transmission

OSv(Driver)

BitVisor(Device)

Req.

PRO/1000

Req.

TX Ring

Req.

Req. Req. Req.

Req.Req.Req.

Saturday, October 18, 14

Packet transmission

OSv(Driver)

BitVisor(Device)

Req.

PRO/1000

Req.

TX Ring

Req.

Req. Req. Req.

Req.Req.Req.

Req.

Saturday, October 18, 14

Packet transmission

OSv(Driver)

BitVisor(Device)

Req.

PRO/1000

Req.

TX Ring

Req.

Req. Req. Req.

Req.Req.Req.

Req.

Notify!(PIO)

Saturday, October 18, 14

Packet transmission

OSv(Driver)

BitVisor(Device)

Req.

PRO/1000

Req.

TX Ring

Req.

Req. Req. Req.

Req.Req.Req.

Req.

Notify!(PIO)

Req.

Req.

Saturday, October 18, 14

Packet transmission

OSv(Driver)

BitVisor(Device)

Req.

PRO/1000

Req.

TX Ring

Req.

Req. Req. Req.

Req.Req.Req.

Req.

Notify!(PIO)

Req.

Req.

Saturday, October 18, 14

Packet transmission

OSv(Driver)

BitVisor(Device)

Req.

PRO/1000

Req.

TX Ring

Req.

Req. Req. Req.

Req.Req.Req.

Req.

Notify!(PIO)

Req.

Req.

Reuse

Saturday, October 18, 14

Packet reception

OSv(Driver)

BitVisor(Device)

Buf.

PRO/1000

Buf.

RX Ring

Buf.

Buf. Buf. Buf.

Buf.Buf.Buf.

Saturday, October 18, 14

Packet reception

OSv(Driver)

BitVisor(Device)

Buf.

PRO/1000

Buf.

RX Ring

Buf.

Buf. Buf. Buf.

Buf.Buf.Buf.

Buf.

Saturday, October 18, 14

Packet reception

OSv(Driver)

BitVisor(Device)

Buf.

PRO/1000

Buf.

RX Ring

Buf.

Buf. Buf. Buf.

Buf.Buf.Buf.

Buf.

Buf.

Saturday, October 18, 14

Packet reception

OSv(Driver)

BitVisor(Device)

Buf.

PRO/1000

Buf.

RX Ring

Buf.

Buf. Buf. Buf.

Buf.Buf.Buf.

Buf.

Buf.

Buf.

Saturday, October 18, 14

Packet reception

OSv(Driver)

BitVisor(Device)

Buf.

PRO/1000

Buf.

RX Ring

Buf.

Buf. Buf. Buf.

Buf.Buf.Buf.

Buf.

Buf.

Buf.

Saturday, October 18, 14

Packet reception

OSv(Driver)

BitVisor(Device)

Buf.

PRO/1000

Buf.

RX Ring

Buf.

Buf. Buf. Buf.

Buf.Buf.Buf.

Buf.

Buf.

Buf.

Int!

Saturday, October 18, 14

Packet reception

OSv(Driver)

BitVisor(Device)

Buf.

PRO/1000

Buf.

RX Ring

Buf.

Buf. Buf. Buf.

Buf.Buf.Buf.

Buf.

Int!

Buf.

Buf.

Receive

Saturday, October 18, 14

InterruptBAR1

(MMIO)MSI-X Message Data:Int. vector number (e.g. 0x26)

VMX event injectionfor virtual interrupts

Saturday, October 18, 14

DHCP succeeded !

Saturday, October 18, 14

Summary &future work

• Summary

• Booting OSv on BitVisor

• Virtio NIC

• Future work

• Further testing

• Virtio BLK/RNG/...

Saturday, October 18, 14

Thank you !

http://www.root.ne.jp/nishide/shs/

Saturday, October 18, 14