内存hotplug支持问题

embeddedlwp

hardware memory RAS feauture: memory mirror

embeddedlwp

回复 3# humjb_1983

OS部分：

ACPI + memory management


   

embeddedlwp

回复 5# humjb_1983

In high-level, here is how ACPI memory hotplug works:

hot-add:

1. ACPI sends a hotplug event to a new ACPI memory device object that is
hot-added.
2. The kernel is notified, and verifies if the new memory device object
has not been attached by any handler yet.
3. The memory handler is called, and obtains a new memory range from the
ACPI memory device object.
4. The memory handler calls add_memory() with the new address range.

The above step 1-4 proceeds automatically within the kernel.  No user
input (nor sysfs interface) is necessary.  Step 2 prevents double adds
and step 3 gets a valid address range from the firmware directly.  Step
4 is basically the same as the "probe" interface, but with all the
verification up front, this step is safe.

hot-remove:

1. ACPI sends a hotplug event to an ACPI memory device object that is
requested to hot-remove.
2. The kernel is notified, and verifies if the memory device object is
attached by a handler.
3. The memory handler is called (which is being attached), and obtains
its memory range.
4. The memory handler calls remove_memory() with the address range.
5. The kernel calls eject method of the ACPI memory device object.


   

embeddedlwp

回复 5# humjb_1983

The memory hotplug add memory consists two steps: online and add
add —— alloc memmap, if we using vmemmap, we also need alloc pagetables for it.
Online ——
The Memory hotplug remove memory consists two steps: offline and remove
offline —— Offline just migrates user pages to somewhere else, and isolates the pages from buddy
system so that no one can use them any more.
remove —— Remove will free kernel direct mapping pagetables, and memmap. If we are using
vmemmap, we also free the pagetables of vmemmap.
lock_memory_hotplug() will protect logical and physical add/removy memory


Logical add memory
lock_memory_hotplug()
Send memory hotplug event MEM_GOING_ONLINE
If zone is not present, setup zone's pcp(next step free pages to buddy will need it)
Online all pages, actually is clear PG_reserved flag, just like early allocator free
pages to buddy system
Update zone's field, manged_pages, present_pages and node's present pages
If zone is not present before, build all zonelists in order that other zone can fallback
to this zone, otherwise update zone's pcp since pcp's batch is acculated according
to zone's present pages and this value change during online pages
update watermark, lowmem reservation, inactive ratio
If the node is not present before, create kswapd
Recaculate total pages
Send memory hotplug event MEM_ONLINE
unlock_memory_hotplug()


Logical remove memory
lock_memory_hotplug()
Making page-allocation-type to be MIGRATE_ISOLATE in order that free pages in the
range will never be allocated.
Send memory hotplug event MEM_GOING_OFFLINE
Memory offline code does 5 times of retry with 120 seconds timeout
Drain pagevec and pageset
Scan range and migrate pages if they are on lru
Test if all pageblock are MIGRATE_ISOLATE type
Offline pages, actually is set PG_reserved flag, then buddy system can't use them
Rechange all pageblocks to MIGRATE_MOVABLE
Update watermark, lowmem reservation, inactive ratio
Build all zonelists f zone is not present any more, otherwise update zone's pcp
If node is not present any more, stop kswapd
Update total memory
unlock_memory_hotplug()


Physical add memory
lock_memory_hotplug()
If node is not present before, hotadd_new_pgdat
Add memory section one by one
Create section decriptor mem_section, add memmap, usemap(bitmap for pageblock migration
type), alloc pagetables and populate them if config sparse-vmemmap
Grow zone/pgdat span
Set PG_reserved flag
Register memory section one by one to memory block
Add firmware memmap entry
unlock_memory_hotplug()


Physical remove memory
Offline memory blocks in range one by one
lock_memory_hotplug()
Check if all memory blocks are offline(logic remove)
Remove firmware memmap entry
Remove memory
Remove sections from memory block one by one
Free memmap, usemap, free pagetables if config sparse-vmemmap
Try to offline node if all cpu/memory are offline
unlock_memory_hotplug()
   

embeddedlwp

回复 5# humjb_1983

Now some severs support a hardware memory RAS feature called memory
mirror, something like RAID1. The mirrored memory devices will be configured
with the same address and host same contents. And you could transparently
hot-remove one of the mirrored memory device without any help from OS.

We could think memory migration as an extension to the memory mirror technology.
The basic flow for memory migration is:
1) Find a spare memory device with enough capacity in the system.
2) OS issues a request to firmware to migrate from source memory device (A)
   to the spare memory device (B).
3) Firmware configures A and B into memory mode, and configure A as master
   and B as slave.
4) Firmware resilver the mirror to synchronize the content from A to B
5) Firmware reconfigure B as master and A as slave.
6) Firmware deconfigures the memory mirror and removes A
7) Firmware report results to OS.
Now user could hot-remove the source memory device A from system.

During memory migration, A and B are in mirror mode, so CPUs and IO devices
could access it as normal. After memory migration, memory device B will have
the same address ranges and content as memory device A, so there's no OS
visible changes except latency (because A and B may belong to different NUMA
domains).

So hardware memory migration could be used to migrate pages can't be migrated
by OS.


   

embeddedlwp

回复 9# 瀚海书香

应该是可软件触发，可硬件触发

   

embeddedlwp

Arrange hotpluggable memory as ZONE_MOVABLE
这个patchset 在缓解这个问题

内核占用的pages不能迁移
huawei通过类似memory mirror的方法实现貌似

memory mirror和memory hotplug实际是两套独立的方案吧？
我的理解是不能说是两套方案，8楼的方法，也就是extend memory mirror就是解决内核占用的pages不能迁移。

embeddedlwp

回复 15# humjb_1983

没资料，都是自己读代码。