Embed Size (px)
DESCRIPTION
EMI INFSO-RI Various catalogues keep information that is related – E.g. LFC keeps info about the content of remote SEs, each one with its catalogue – A change in the permissions of a file in LFC is not automatically reflected by the peripheric catalogue – If a SE looses a file, the LFC des not know Keeping them in sync is a very hard problem 13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation The problem
Citation preview
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
Catalogue synchronization&
ACL propagation
Fabrizio Furano (CERN IT-GT)
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
F.Furano - Catalogue Synchronization & ACL propagation
• The problem• The Working Group• Goal and milestones• Message brokers• Communication protocols and guidelines• Conclusions
13 Oct 2010
Outline
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
• Various catalogues keep information that is related– E.g. LFC keeps info about the content of remote
SEs, each one with its catalogue– A change in the permissions of a file in LFC is
not automatically reflected by the peripheric catalogue
– If a SE looses a file, the LFC des not know• Keeping them in sync is a very hard problem
13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation
The problem
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
• Actually, the updates to the various catalogues are performed as external actions– By the jobs, as long as they do not die/crash
in the wrong moment– As scheduled rounds of massive
synchronizations
13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation
The problem
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
F.Furano - Catalogue Synchronization & ACL propagation
• Current consistency model is not resilient to failures– Storage failures lead to dangling entries to be
cleaned up manually. Catalogue failures lead to orphaned files.
• Namespace scanning for diffs is an expensive workaround
13 Oct 2010
The problem
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
F.Furano - Catalogue Synchronization & ACL propagation
• Make the various catalogues/SE able to talk to each other– In order to exchange messages that keep
them synchronized– 2 ways:
• LFC->SE to propagate changes in the permissions• SE->LFC to propagate info about lost files
– These are just starting examples, what can be done can be even much more interesting
13 Oct 2010
The proposal
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
F.Furano - Catalogue Synchronization & ACL propagation
• In other words:– Disasters and misalignments will still happen
• Nobody can prevent a file from disappearing– Let’s design a mechanism that ruggedizes the
catalogues by means of notifications– Let’s design a mechanism to propagate the
changes in the LFC to all the interested SEs
13 Oct 2010
The proposal
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
F.Furano - Catalogue Synchronization & ACL propagation
• The kickoff mail was sent a few days ago in emi-jra1-data
• Working group:– Jean-Philippe B.– Paul M.– Riccardo Z.– Michele D.– Fabrizio F. (Me)
13 Oct 2010
The working group
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
F.Furano - Catalogue Synchronization & ACL propagation
• Proposed demonstrators to use reliable message (i.e. industry standard MQ) as backbone of the reliability– All interested catalogues can “subscribe” for
permissions that changed in the LFC– Lost files can be broadcast on the “lost” topic
to interested catalogues• Somehow possible also for corrupted “bad” files
(e.g. not readable)
13 Oct 2010
Milestone
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
F.Furano - Catalogue Synchronization & ACL propagation
Looking for good ways to reliably communicate and cooperate
Communication
13 Oct 2010
SE1 SEnSE2
SE or exp. catalogue
subscribes to the relevant topics
(e.g. “Changes”)
SE sends to the appropriate topics (e.g.
“Lost”)
LFC Subscribes to the relevant
topics (e.g. “Lost”)
SE Sends to the appropriate topics (e.g. “Changes”)
LFC
ATLAScatalogueBroker(s)
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
F.Furano - Catalogue Synchronization & ACL propagation
• Industry standard MQ brokers, like ActiveMQ will constitute the backbone
• Need to:– Agree on the needed messages
(content+semantics)– Decide the implementation guidelines
13 Oct 2010
Message brokers
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
F.Furano - Catalogue Synchronization & ACL propagation
• Several APIs are available, usable from nearly every language
• Each API talks to the broker using a protocol. Mostly, in our case:– OpenWire (binary)– STOMP (XML based)
• The broker speaks them all and assures the interoperability
• The idea is not to enforce one protocol, just give usage guidelines
13 Oct 2010
APIs and protocols
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
F.Furano - Catalogue Synchronization & ACL propagation
• Although compatible, each API/Protocol has its own idiosyncrasies. E.g.:
• The recent release of the C++ library is simply broken. I had to do all the tests with (their) trunk, with very good results.
• The reference STOMP implementation that I tested does not support recovery. Any glitch silently makes a zombie client until manual restart.
• Probably many others.– Need to sort out the needed features
13 Oct 2010
APIs and protocols
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
F.Furano - Catalogue Synchronization & ACL propagation
• Many aspects have still to be sorted out (E.g. security requirements), but at least we started well.
• Making catalogues and SEs interact seems a good way to solve the consistency problem
• The messaging broker tech is not new, at least we know that it works
• The performance seems good, eventually it should also be sufficiently scalable
• Next goal: demo for December 2010
13 Oct 2010
Conclusions
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
Thank you
13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation
EMI is partially funded by the European Commission under Grant Agreement INFSO-RI-261611
