This is the plan for image cache and proxy further development before it gets into master
Document the whole thing
Add articles for newly introduced actions and options to the CLI page. Also, it would be good to have an article describing the protocols involved.
Make the unix sockets reside in work-dir
Looking at do_open_remote_image() I see that it fchdir-s to image dir before connecting to proxy/cache. Better solution is to put the socket into workdir.
After this the option --images-dir should become optional. Provided the --remote is given CRIU should work purely on the work-dir and not generate anything in the images-dir.
Tune up the image_cache and image_proxy commands to obey daemon CLI rules
They should accept the --status-fd and --pidfile options. Presumably the very cr_daemon() call should be equipped with everything that should be done for CLI/opt/--daemon and proxy/cache tasks should just call it :)
Fix local connections not to generate per-image threads
There can be many images and it's not nice to stress the system with such amount of threads. The criu/uffd.c manages multiple descriptors with page-faults using the epoll stuff and can be used as an example.
The accept_remote_image_connections() seem not to work well with opts.ps_socket RPC as the former just calls accept() on whatever socket is passed there, while the opts.ps_socket is already an established socket for data transfer.
No strings in protocol
Now the hard-coded "RESTORE_FINISH" string (and DUMP_FINISHED one) is used to terminate the communication. Need to tune up the protobuf objects to send boolean (or integer) EOF sign rather that the string.
Check how proxy/cache works with incremental dumps
Looking at the skip_remote_bytes() I think that image-cache and -proxy still do not work well with stacked pages images. Probably for those we'll need the page server or lazy pages like protocol that would request the needed regions and receive it back rather than read bytes from sockets simply to skip those.
Add support for cache/proxy into go-phaul code
I haven't yet finished with the prototype, but plan to do it soon, so once the above steps are done we'll be able to proceed with this one.