bugThe GNU Hurd - Bugs: bug #28730, Bad Mach object cache reuse

 
 

bug #28730: Bad Mach object cache reuse

Submitted by:  Samuel Thibault <sthibaul>
Submitted on:  Sun 24 Jan 2010 10:28:15 PM UTC  
 
Category: MetaSeverity: 3 - Normal
Priority: 5 - NormalItem Group: None
Status: NonePrivacy: Public
Assigned to: NoneOriginator Name: 
Open/Closed: OpenReproducibility: None
Size (loc): NonePlanned Release: None
Effort: 0.00
Wiki-like text discussion box: 

Pagers only flush pages from their memory objects in special situations like when truncating/removing a node, to be able to use them as cache. So, for most cases, resident_page_count will be > 0.

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Mon 05 Sep 2011 11:38:12 PM UTC, comment #2:

Here is a patch I had been experimenting with, which should simply drop
the limit, and let the pageout daemon eliminate objects as their pages
get dropped. It however leads Debian buildds to run out of memory,
apparently the daemon does not drop pages fast enough?

(file #23951)

Samuel Thibault <sthibaul>
Project Member
Fri 30 Apr 2010 12:03:13 AM UTC, comment #1:

I think we should replace the cache system for memory objects (at least for the ones with external pagers). How about something like this:

  • GNU Mach
    • Get rid of vm_object_cached_*. If an object can persist, it'll persist forever (or until a pager changes its attributes).
    • Separate pages of memory objects with external pagers into a different queue. Treat it like cache. If we want to free some, ask the translator (by using one of its pagers) to unallocate some objects (translators can select the ones they want to, Mach will only give them a hint on how many space should be freed). This requires a new interface.
    • (Optional) Maintain cached pages statistics per task (pager's owner). Use it as foundation for cache quotas.
  • Hurd
    • Keep track of every action that could alter the contents of a pager. Take special attention with mmap(). Use a lru queue with pointers to the pagers. For everyone of those actions, mark the object as dirty and update queue.
    • Sync only objects marked as dirty.
    • When GNU Mask ask us to unallocate some objects, start running through the queue until the hint provided by the kernel is archieved.
Sergio López <k0ro>
Project Member
Sun 24 Jan 2010 10:28:15 PM UTC, original submission:

/var/tmp/hurd-20090404 holds 4142 files totalling 77MB
/var/tmp/sudo* holds 434 files totalling 26MB

vm_object_cached_max is set to 4000 in my gnumach, thus not enough for
the whole hurd-20090404

I have 256MB memory, thus plenty for cache

$ time rgrep foobarbaz /var/tmp/hurd-20090404
1m5
$ time rgrep foobarbaz /var/tmp/sudo*
15s
$ time rgrep foobarbaz /var/tmp/sudo*
10s
$ time rgrep foobarbaz /var/tmp/sudo*
8s
$ time rgrep foobarbaz /var/tmp/sudo*
5s
$ time rgrep foobarbaz /var/tmp/sudo*
4s
$ reboot

$ time rgrep foobarbaz /var/tmp/sudo*
15s
$ time rgrep foobarbaz /var/tmp/sudo*
4s

This means filling the cache with hurd-20090404 first gets in the way
of getting sudo* into the cache.

I've dug a bit, the problem lies in the periodic sync of ext2fs. If I
pass --sync=60 to ext2fs, and then retry

$ time rgrep foobarbaz /var/tmp/hurd-20090404
1m5
$ ... wait for ext2fs sync to happen, then as soon as it's over,
$ time rgrep foobarbaz /var/tmp/sudo*
15s
$ time rgrep foobarbaz /var/tmp/sudo*
4s

What I can notice is that on ext2fs sync, all its cached objects get
out of the object cache (for the sync) and then put again (probably
because of the call to memory_object_lock_request()). In the first
scenario, it means the following happening on the object cache (put in
brackets, objects on the left were queued first).

[]
$ time rgrep foobarbaz /var/tmp/hurd-20090404
[ ---------------- hurd-20090404 objs ------------------- ]
$ time rgrep foobarbaz /var/tmp/sudo*
[ hurd-20090404 objs ------------------------ | sudo objs ]
# ext2fs sync
[ ------ mixture of hurd-20090404 objs and sudo objs ---- ]
# rgrep continues, new sudo objs push the mixture out, and some sudo
# objs get dropped
[ mixture of hurd-20090404 objs and sudo objs | sudo objs ]
# ext2fs sync
[ ------ mixture of hurd-20090404 objs and sudo objs ---- ]
[ mixture of hurd-20090404 objs and sudo objs | sudo objs ]
15s

I.e. the ext2fs sync mixes old useless objects with sudo objs,
according to hash functions (see diskfs_node_iterate() or
ports_bucket_iterate()), and in the end a lot of the sudo objs have
been pushed out and dropped from the cache, which is unfortunate. When
raising the period to 60s, the rgrep has the time to finish before its
sudo objects gets scrambled with the hurd objects, and then cache hit
can happen.

To summarize, the issue is that write_all_disknodes() touches the
cached objects too heavily so that GNU Mach believes they are being
used, while they're not. Raising vm_object_cached_max is not a
solution since that won't prevent the scrambling, just reduce its
effect. Tinkering with memory_object_lock_request() is not so easy
either, the ground issue being that ext2fs issues syncs for all objects
in a basically random order. Could there be a way for ext2fs to know
which objects need a sync?

Samuel Thibault <sthibaul>
Project Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #23951:  patch-cache added by sthibaul (2kB - application/octet-stream - experimental patch to simply drop the limit)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by k0ro (Posted a comment)
  • -unavailable- added by sthibaul (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 2 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 07 Sep 2011 06:44:55 PM UTCk0roWiki-like text discussion box=>Pagers only flush pages from their memory objects in special situations like when truncating/removing a node, to be able to use them as cache. So, for most cases, resident_page_count will be > 0.
    Mon 05 Sep 2011 11:38:12 PM UTCsthibaulAttached File-=>Added patch-cache, #23951

    Back to the top


    Powered by Savane 3.1-cleanup