内存hotplug支持问题

humjb_1983

根据目前了解，Linux对于CPU的热拔操作还不能完全支持，主要问题在于内核自身占用的内存不能热拔，而内核自己占用的内存可能分布于所有的内存节点上，可能导致所有的内存条都不能拔。
最新内核中，针对这样的情况提供了一些补丁，主要是想将内核自己占用的内存限制在同一个节点中，以便于可以热拔其他的内存节点，但是这样的话，由于NUMA远程节点访问效率的问题，会导致系统性能下降，而且在内核初始化的时候的部分内存还不能限制，所以，目前来看Linux要支持内存热拔，问题还比较多。
不知是否有专家做过相关优化和研究，欢迎探讨。
另外，据了解，HW的服务器号称可以支持内存的热插拔(也是Linux OS)，不知是否有兄弟了解相关的实现原理，望赐教~~

embeddedlwp

hardware memory RAS feauture: memory mirror

humjb_1983

embeddedlwp 发表于 2013-12-18 17:50
hardware memory RAS feauture: memory mirror

呵呵，感谢！
你说的这个内存镜像功能实际是完全由硬件实现，即由硬件发起，由BIOS完成内存迁移操作，迁移前后的物理地址空间是不变的，整个过程对OS来说都是透明的。
所以，这里的内存镜像功能其实跟OS中的memory hotplug没有直接的关系，OS的内存hotplug是指由硬件发起，通过中断通知OS(和BIOS)，由OS进行内存迁移(将问题内存条上的页面迁移到其他空闲物理内存中，同时修改相应的页表)，迁移后物理地址空间是会发生变化的，这个是需要软硬件配合实现的。

embeddedlwp

回复 3# humjb_1983

OS部分：

ACPI + memory management


   

humjb_1983

embeddedlwp 发表于 2013-12-19 09:14
回复 3# humjb_1983

OS部分：

呵呵，不是很明白OS部分具体需要做什么？能否明示？

embeddedlwp

回复 5# humjb_1983

In high-level, here is how ACPI memory hotplug works:

hot-add:

1. ACPI sends a hotplug event to a new ACPI memory device object that is
hot-added.
2. The kernel is notified, and verifies if the new memory device object
has not been attached by any handler yet.
3. The memory handler is called, and obtains a new memory range from the
ACPI memory device object.
4. The memory handler calls add_memory() with the new address range.

The above step 1-4 proceeds automatically within the kernel.  No user
input (nor sysfs interface) is necessary.  Step 2 prevents double adds
and step 3 gets a valid address range from the firmware directly.  Step
4 is basically the same as the "probe" interface, but with all the
verification up front, this step is safe.

hot-remove:

1. ACPI sends a hotplug event to an ACPI memory device object that is
requested to hot-remove.
2. The kernel is notified, and verifies if the memory device object is
attached by a handler.
3. The memory handler is called (which is being attached), and obtains
its memory range.
4. The memory handler calls remove_memory() with the address range.
5. The kernel calls eject method of the ACPI memory device object.


   

embeddedlwp

回复 5# humjb_1983

The memory hotplug add memory consists two steps: online and add
add —— alloc memmap, if we using vmemmap, we also need alloc pagetables for it.
Online ——
The Memory hotplug remove memory consists two steps: offline and remove
offline —— Offline just migrates user pages to somewhere else, and isolates the pages from buddy
system so that no one can use them any more.
remove —— Remove will free kernel direct mapping pagetables, and memmap. If we are using
vmemmap, we also free the pagetables of vmemmap.
lock_memory_hotplug() will protect logical and physical add/removy memory


Logical add memory
lock_memory_hotplug()
Send memory hotplug event MEM_GOING_ONLINE
If zone is not present, setup zone's pcp(next step free pages to buddy will need it)
Online all pages, actually is clear PG_reserved flag, just like early allocator free
pages to buddy system
Update zone's field, manged_pages, present_pages and node's present pages
If zone is not present before, build all zonelists in order that other zone can fallback
to this zone, otherwise update zone's pcp since pcp's batch is acculated according
to zone's present pages and this value change during online pages
update watermark, lowmem reservation, inactive ratio
If the node is not present before, create kswapd
Recaculate total pages
Send memory hotplug event MEM_ONLINE
unlock_memory_hotplug()


Logical remove memory
lock_memory_hotplug()
Making page-allocation-type to be MIGRATE_ISOLATE in order that free pages in the
range will never be allocated.
Send memory hotplug event MEM_GOING_OFFLINE
Memory offline code does 5 times of retry with 120 seconds timeout
Drain pagevec and pageset
Scan range and migrate pages if they are on lru
Test if all pageblock are MIGRATE_ISOLATE type
Offline pages, actually is set PG_reserved flag, then buddy system can't use them
Rechange all pageblocks to MIGRATE_MOVABLE
Update watermark, lowmem reservation, inactive ratio
Build all zonelists f zone is not present any more, otherwise update zone's pcp
If node is not present any more, stop kswapd
Update total memory
unlock_memory_hotplug()


Physical add memory
lock_memory_hotplug()
If node is not present before, hotadd_new_pgdat
Add memory section one by one
Create section decriptor mem_section, add memmap, usemap(bitmap for pageblock migration
type), alloc pagetables and populate them if config sparse-vmemmap
Grow zone/pgdat span
Set PG_reserved flag
Register memory section one by one to memory block
Add firmware memmap entry
unlock_memory_hotplug()


Physical remove memory
Offline memory blocks in range one by one
lock_memory_hotplug()
Check if all memory blocks are offline(logic remove)
Remove firmware memmap entry
Remove memory
Remove sections from memory block one by one
Free memmap, usemap, free pagetables if config sparse-vmemmap
Try to offline node if all cpu/memory are offline
unlock_memory_hotplug()
   

embeddedlwp

回复 5# humjb_1983

Now some severs support a hardware memory RAS feature called memory
mirror, something like RAID1. The mirrored memory devices will be configured
with the same address and host same contents. And you could transparently
hot-remove one of the mirrored memory device without any help from OS.

We could think memory migration as an extension to the memory mirror technology.
The basic flow for memory migration is:
1) Find a spare memory device with enough capacity in the system.
2) OS issues a request to firmware to migrate from source memory device (A)
   to the spare memory device (B).
3) Firmware configures A and B into memory mode, and configure A as master
   and B as slave.
4) Firmware resilver the mirror to synchronize the content from A to B
5) Firmware reconfigure B as master and A as slave.
6) Firmware deconfigures the memory mirror and removes A
7) Firmware report results to OS.
Now user could hot-remove the source memory device A from system.

During memory migration, A and B are in mirror mode, so CPUs and IO devices
could access it as normal. After memory migration, memory device B will have
the same address ranges and content as memory device A, so there's no OS
visible changes except latency (because A and B may belong to different NUMA
domains).

So hardware memory migration could be used to migrate pages can't be migrated
by OS.


   

瀚海书香

回复 1# humjb_1983
貌似国内富士通南大搞这个的比较多。
应该也是软件触发，将某一个node的数据移到其余node上，然后把这个node物理移除。

   

embeddedlwp

回复 9# 瀚海书香

应该是可软件触发，可硬件触发