Transferring near infrared spectroscopic calibration model across different harvested seasons using joint distribution adaptation

Near infrared spectroscopic (NIRS) data from different harvested seasons may consist of different feature spaces even though the samples have the same label values. This is because the spectral response could be affected by the changes in environmental parameters, internal quality, and the reproducibility of NIRS instruments. Thus, this study aims to investigate the ability of Joint Distribution Adaptation (JDA) transfer learning algorithm in addressing the assumption of traditional machine learning i.e. both training and testing data must come from the same feature spaces and data distribution. First, NIRS data acquired from two different harvested seasons were used as the source domain and the target domain, respectively. Next, JDA was implemented to produce an adaptation matrix using the source domain and transfer datasets. This adaptation matrix would be used to transform the source and target domain datasets. After that, a calibration model was developed by means of Partial Least Squares (PLS) using the transformed training dataset; and validated using the trans�formed independent testing dataset. The proposed JDA-PLS was compared to the PLS without transfer learning as the baseline learning. Findings show that the proposed JDA-PLS with 10 LVs achieved the lowest RMSEP of 1.134% and the highest RP 2 of 0.826

공동 대조적 학습을 이용한 비지도 도메인 적응 기법 연구

학위논문 (석사) -- 서울대학교 대학원 : 공과대학 전기·정보공학부, 2021. 2. 윤성로.Domain adaptation is introduced to exploit the label information of source domain when labels are not available for target domain. Previous methods minimized domain discrepancy in a latent space to enable transfer learning. These studies are based on the theoretical analysis that the target error is upper bounded by the sum of source error, the domain discrepancy, and the joint error of the ideal hypothesis. However, feature discriminability is sacrificed while enhancing the feature transferability by matching marginal distributions. In particular, the ideal joint hypothesis error in the target error upper bound, which was previously considered to be minute, has been found to be significant, impairing its theoretical guarantee. In this paper, to manage the joint error, we propose an alternative upper bound on the target error that explicitly considers it. Based on the theoretical analysis, we suggest a joint optimization framework that combines the source and target domains. To minimize the joint error, we further introduce Joint Contrastive Learning (JCL) that finds class-level discriminative features. With a solid theoretical framework, JCL employs contrastive loss to maximize the mutual information between a feature and its label, which is equivalent to maximizing the Jensen-Shannon divergence between conditional distributions. Extensive experiments on domain adaptation datasets demonstrate that JCL outperforms existing state-of-the-art methods.도메인 적응 기법은 타겟 도메인의 라벨 정보가 없는 상황에서 비슷한 도메인인 소스 도메인의 라벨 정보를 활용하기 위해 개발되었다. 기존의 방법론들은 잠재 공간에서 도메인들 사이의 분포 차이를 줄임으로써 전이 학습이 가능하게 하였다. 이러한 기법들은 소스 도메인의 에러율, 도메인 간 분포 차이, 그리고 양 도메인에서 이상적인 분류기의 에러율의 합이 타겟 도메인의 에러율의 상계가 된다는 이론을 바탕으로 한다. 그러나 도메인들 사이의 분포 차이를 줄이는 방법들은 동시에 잠재 공간에서 서로 다른 라벨을 갖는 데이터들 사이의 구별성을 감소시켰다. 특히, 작을 것이라 생각되던 양 도메인에서 이상적인 분류기의 에러율이 큰 것으로 나타났다. 본 논문에서는 기존의 이론에서는 다루지 않은 양 도메인에서 분류기의 에러율을 조절할 수 있게하기 위해 새로운 이론을 제시한다. 이 이론적 배경을 바탕으로 소스 도메인과 타겟 도메인을 함께 학습하는 공동 대조적 방법을 소개한다. 본 공동 대조적 학습 방법에서는 각 라벨별로 구분되는 잠재 공간을 학습하기 위해 각 데이터의 특징과 라벨 사이의 상호 정보량을 최대화한다. 이 각 데이터의 특징과 라벨 사이의 상호 정보량은 각 라벨 분포 사이의 젠센-샤논 거리와 같으므로 이를 최대화하는 것은 곧 라벨들이 잘 구별되는 잠재 공간을 학습하는 것이다. 마지막으로 공동 대조적 학습 방법을 여러 데이터 셋에 적용하여 기존 방법론들과 비교하였다.1 Introduction 1 2 Background 4 2.1 Domain Adaptation . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.1.1 Problem Setting and Notations . . . . . . . . . . . . . . . . . 4 2.1.2 Theoretical Background . . . . . . . . . . . . . . . . . . . . 5 2.2 Approaches for Domain Adaptation . . . . . . . . . . . . . . . . . . 6 2.2.1 Marginal Distribution Alignment Based Approaches . . . . . 6 2.2.2 Conditional Distribution Matching Approaches . . . . . . . . 7 2.3 Contrastive Learning . . . . . . . . . . . . . . . . . . . . . . . . . . 8 3 Method 10 3.1 An Alternative Upper Bound . . . . . . . . . . . . . . . . . . . . . . 10 3.2 Joint Contrastive Learning . . . . . . . . . . . . . . . . . . . . . . . 14 3.2.1 Theoretical Guarantees . . . . . . . . . . . . . . . . . . . . . 14 3.2.2 Generalization to Multiclass Classification . . . . . . . . . . 17 3.2.3 Training Procedure . . . . . . . . . . . . . . . . . . . . . . . 19 4 Experiments 24 4.1 Datasets and Baselines . . . . . . . . . . . . . . . . . . . . . . . . . 24 4.2 Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . 26 4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 4.4 Ablation Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 5 Conclusion 35 Abstract (In Korean) 45Maste

A review of domain adaptation without target labels

Domain adaptation has become a prominent problem setting in machine learning and related fields. This review asks the question: how can a classifier learn from a source domain and generalize to a target domain? We present a categorization of approaches, divided into, what we refer to as, sample-based, feature-based and inference-based methods. Sample-based methods focus on weighting individual observations during training based on their importance to the target domain. Feature-based methods revolve around on mapping, projecting and representing features such that a source classifier performs well on the target domain and inference-based methods incorporate adaptation into the parameter estimation procedure, for instance through constraints on the optimization procedure. Additionally, we review a number of conditions that allow for formulating bounds on the cross-domain generalization error. Our categorization highlights recurring ideas and raises questions important to further research.Comment: 20 pages, 5 figure

Joint Distribution Optimal Transportation for Domain Adaptation

This paper deals with the unsupervised domain adaptation problem, where one wants to estimate a prediction function $f$ in a given target domain without any labeled sample by exploiting the knowledge available from a source domain where labels are known. Our work makes the following assumption: there exists a non-linear transformation between the joint feature/label space distributions of the two domain $\mathcal{P}_s$ and $\mathcal{P}_t$. We propose a solution of this problem with optimal transport, that allows to recover an estimated target $\mathcal{P}^f_t=(X,f(X))$ by optimizing simultaneously the optimal coupling and $f$. We show that our method corresponds to the minimization of a bound on the target error, and provide an efficient algorithmic solution, for which convergence is proved. The versatility of our approach, both in terms of class of hypothesis or loss functions is demonstrated with real world classification and regression problems, for which we reach or surpass state-of-the-art results.Comment: Accepted for publication at NIPS 201