2,549 research outputs found
Recommended from our members
A semantic web services-based infrastructure for context-adaptive process support
Current technologies aimed at supporting processes whether it is a business or learning process - primarily follow a metadata- and data-centric paradigm. Whereas process metadata is usually based on a specific standard specification - such as the Business Process Modeling Notation (BPMN) or the IMS Learning Design Standard - the allocation of resources is done manually at design-time, and the used data is often specific to one process context only. These facts limit the reusability of process models across different standards and contexts. To overcome these issues, we introduce an innovative Semantic Web Service-based framework aimed at changing the current paradigm to a context-adaptive service-oriented approach. Following the idea of layered semantic abstractions, our approach supports the development of abstract semantic process model - reusable across different contexts and standards - that enables a dynamic adaptation to specific actor needs and objectives. To illustrate the application of our framework and establish its feasibility, we describe a prototypical application in the E-Learning domain
Multimodal Federated Learning via Contrastive Representation Ensemble
With the increasing amount of multimedia data on modern mobile systems and
IoT infrastructures, harnessing these rich multimodal data without breaching
user privacy becomes a critical issue. Federated learning (FL) serves as a
privacy-conscious alternative to centralized machine learning. However,
existing FL methods extended to multimodal data all rely on model aggregation
on single modality level, which restrains the server and clients to have
identical model architecture for each modality. This limits the global model in
terms of both model complexity and data capacity, not to mention task
diversity. In this work, we propose Contrastive Representation Ensemble and
Aggregation for Multimodal FL (CreamFL), a multimodal federated learning
framework that enables training larger server models from clients with
heterogeneous model architectures and data modalities, while only communicating
knowledge on public dataset. To achieve better multimodal representation
fusion, we design a global-local cross-modal ensemble strategy to aggregate
client representations. To mitigate local model drift caused by two
unprecedented heterogeneous factors stemming from multimodal discrepancy
(modality gap and task gap), we further propose two inter-modal and intra-modal
contrasts to regularize local training, which complements information of the
absent modality for uni-modal clients and regularizes local clients to head
towards global consensus. Thorough evaluations and ablation studies on
image-text retrieval and visual question answering tasks showcase the
superiority of CreamFL over state-of-the-art FL methods and its practical
value.Comment: ICLR 2023. Code is available at https://github.com/FLAIR-THU/CreamF
FedMEKT: Distillation-based Embedding Knowledge Transfer for Multimodal Federated Learning
Federated learning (FL) enables a decentralized machine learning paradigm for
multiple clients to collaboratively train a generalized global model without
sharing their private data. Most existing works simply propose typical FL
systems for single-modal data, thus limiting its potential on exploiting
valuable multimodal data for future personalized applications. Furthermore, the
majority of FL approaches still rely on the labeled data at the client side,
which is limited in real-world applications due to the inability of
self-annotation from users. In light of these limitations, we propose a novel
multimodal FL framework that employs a semi-supervised learning approach to
leverage the representations from different modalities. Bringing this concept
into a system, we develop a distillation-based multimodal embedding knowledge
transfer mechanism, namely FedMEKT, which allows the server and clients to
exchange the joint knowledge of their learning models extracted from a small
multimodal proxy dataset. Our FedMEKT iteratively updates the generalized
global encoders with the joint embedding knowledge from the participating
clients. Thereby, to address the modality discrepancy and labeled data
constraint in existing FL systems, our proposed FedMEKT comprises local
multimodal autoencoder learning, generalized multimodal autoencoder
construction, and generalized classifier learning. Through extensive
experiments on three multimodal human activity recognition datasets, we
demonstrate that FedMEKT achieves superior global encoder performance on linear
evaluation and guarantees user privacy for personal data and model parameters
while demanding less communication cost than other baselines
Trust, Accountability, and Autonomy in Knowledge Graph-based AI for Self-determination
Knowledge Graphs (KGs) have emerged as fundamental platforms for powering
intelligent decision-making and a wide range of Artificial Intelligence (AI)
services across major corporations such as Google, Walmart, and AirBnb. KGs
complement Machine Learning (ML) algorithms by providing data context and
semantics, thereby enabling further inference and question-answering
capabilities. The integration of KGs with neuronal learning (e.g., Large
Language Models (LLMs)) is currently a topic of active research, commonly named
neuro-symbolic AI. Despite the numerous benefits that can be accomplished with
KG-based AI, its growing ubiquity within online services may result in the loss
of self-determination for citizens as a fundamental societal issue. The more we
rely on these technologies, which are often centralised, the less citizens will
be able to determine their own destinies. To counter this threat, AI
regulation, such as the European Union (EU) AI Act, is being proposed in
certain regions. The regulation sets what technologists need to do, leading to
questions concerning: How can the output of AI systems be trusted? What is
needed to ensure that the data fuelling and the inner workings of these
artefacts are transparent? How can AI be made accountable for its
decision-making? This paper conceptualises the foundational topics and research
pillars to support KG-based AI for self-determination. Drawing upon this
conceptual framework, challenges and opportunities for citizen
self-determination are illustrated and analysed in a real-world scenario. As a
result, we propose a research agenda aimed at accomplishing the recommended
objectives
Bidirectional Contrastive Split Learning for Visual Question Answering
Visual Question Answering (VQA) based on multi-modal data facilitates
real-life applications such as home robots and medical diagnoses. One
significant challenge is to devise a robust decentralized learning framework
for various client models where centralized data collection is refrained due to
confidentiality concerns. This work aims to tackle privacy-preserving VQA by
decoupling a multi-modal model into representation modules and a contrastive
module and leveraging inter-module gradients sharing and inter-client weight
sharing. To this end, we propose Bidirectional Contrastive Split Learning
(BiCSL) to train a global multi-modal model on the entire data distribution of
decentralized clients. We employ the contrastive loss that enables a more
efficient self-supervised learning of decentralized modules. Comprehensive
experiments are conducted on the VQA-v2 dataset based on five SOTA VQA models,
demonstrating the effectiveness of the proposed method. Furthermore, we inspect
BiCSL's robustness against a dual-key backdoor attack on VQA. Consequently,
BiCSL shows much better robustness to the multi-modal adversarial attack
compared to the centralized learning method, which provides a promising
approach to decentralized multi-modal learning
Context-adaptive learning designs by using semantic web services
IMS Learning Design (IMS-LD) is a promising technology aimed at supporting learning processes. IMS-LD packages contain the learning process metadata as well as the learning resources. However, the allocation of resources - whether data or services - within the learning design is done manually at design-time on the basis of the subjective appraisals of a learning designer. Since the actual learning context is known at runtime only, IMS-LD applications cannot adapt to a specific context or learner. Therefore, the reusability is limited and high development costs have to be taken into account to support a variety of contexts. To overcome these issues, we propose a highly dynamic approach based on Semantic Web Services (SWS) technology. Our aim is moving from the current data- and metadata-based to a context-adaptive service-orientated paradigm We introduce semantic descriptions of a learning process in terms of user objectives (learning goals) to abstract from any specific metadata standards and used learning resources. At runtime, learning goals are accomplished by automatically selecting and invoking the services that fit the actual user needs and process contexts. As a result, we obtain a dynamic adaptation to different contexts at runtime. Semantic mappings from our standard-independent process models will enable the automatic development of versatile, reusable IMS-LD applications as well as the reusability across multiple metadata standards. To illustrate our approach, we describe a prototype application based on our principles
Balancing Privacy Protection and Interpretability in Federated Learning
Federated learning (FL) aims to collaboratively train the global model in a
distributed manner by sharing the model parameters from local clients to a
central server, thereby potentially protecting users' private information.
Nevertheless, recent studies have illustrated that FL still suffers from
information leakage as adversaries try to recover the training data by
analyzing shared parameters from local clients. To deal with this issue,
differential privacy (DP) is adopted to add noise to the gradients of local
models before aggregation. It, however, results in the poor performance of
gradient-based interpretability methods, since some weights capturing the
salient region in feature map will be perturbed. To overcome this problem, we
propose a simple yet effective adaptive differential privacy (ADP) mechanism
that selectively adds noisy perturbations to the gradients of client models in
FL. We also theoretically analyze the impact of gradient perturbation on the
model interpretability. Finally, extensive experiments on both IID and Non-IID
data demonstrate that the proposed ADP can achieve a good trade-off between
privacy and interpretability in FL
Challenges and Remedies to Privacy and Security in AIGC: Exploring the Potential of Privacy Computing, Blockchain, and Beyond
Artificial Intelligence Generated Content (AIGC) is one of the latest
achievements in AI development. The content generated by related applications,
such as text, images and audio, has sparked a heated discussion. Various
derived AIGC applications are also gradually entering all walks of life,
bringing unimaginable impact to people's daily lives. However, the rapid
development of such generative tools has also raised concerns about privacy and
security issues, and even copyright issues in AIGC. We note that advanced
technologies such as blockchain and privacy computing can be combined with AIGC
tools, but no work has yet been done to investigate their relevance and
prospect in a systematic and detailed way. Therefore it is necessary to
investigate how they can be used to protect the privacy and security of data in
AIGC by fully exploring the aforementioned technologies. In this paper, we
first systematically review the concept, classification and underlying
technologies of AIGC. Then, we discuss the privacy and security challenges
faced by AIGC from multiple perspectives and purposefully list the
countermeasures that currently exist. We hope our survey will help researchers
and industry to build a more secure and robust AIGC system.Comment: 43 pages, 10 figure
- …