Greedy Shallow Networks: An Approach for Constructing and Training Neural Networks

We present a greedy-based approach to construct an efficient single hidden layer neural network with the ReLU activation that approximates a target function. In our approach we obtain a shallow network by utilizing a greedy algorithm with the prescribed dictionary provided by the available training data and a set of possible inner weights. To facilitate the greedy selection process we employ an integral representation of the network, based on the ridgelet transform, that significantly reduces the cardinality of the dictionary and hence promotes feasibility of the greedy selection. Our approach allows for the construction of efficient architectures which can be treated either as improved initializations to be used in place of random-based alternatives, or as fully-trained networks in certain cases, thus potentially nullifying the need for backpropagation training. Numerical experiments demonstrate the tenability of the proposed concept and its advantages compared to the conventional techniques for selecting architectures and initializations for neural networks

Advancing the Standards for Unmanned Air System Communications, Navigation and Surveillance

Under NASA program NNA16BD84C, new architectures were identified and developed for supporting reliable and secure Communications, Navigation and Surveillance (CNS) needs for Unmanned Air Systems (UAS) operating in both controlled and uncontrolled airspace. An analysis of architectures for the two categories of airspace and an implementation technology readiness analysis were performed. These studies produced NASA reports that have been made available in the public domain and have been briefed in previous conferences. We now consider how the products of the study are influencing emerging directions in the aviation standards communities. The International Civil Aviation Organization (ICAO) Communications Panel (CP), Working Group I (WG-I) is currently developing a communications network architecture known as the Aeronautical Telecommunications Network with Internet Protocol Services (ATN/IPS). The target use case for this service is secure and reliable Air Traffic Management (ATM) for manned aircraft operating in controlled airspace. However, the work is more and more also considering the emerging class of airspace users known as Remotely Piloted Aircraft Systems (RPAS), which refers to certain UAS classes. In addition, two Special Committees (SCs) in the Radio Technical Commission for Aeronautics (RTCA) are developing Minimum Aviation System Performance Standards (MASPS) and Minimum Operational Performance Standards (MOPS) for UAS. RTCA SC-223 is investigating an Internet Protocol Suite (IPS) and AeroMACS aviation data link for interoperable (INTEROP) UAS communications. Meanwhile, RTCA SC-228 is working to develop Detect And Avoid (DAA) equipment and a Command and Control (C2) Data Link MOPS establishing LBand and C-Band solutions. These RTCA Special Committees along with ICAO CP WG/I are therefore overlapping in terms of the Communication, Navigation and Surveillance (CNS) alternatives they are seeking to provide for an integrated manned- and unmanned air traffic management service as well as remote pilot command and control. This paper presents UAS CNS architecture concepts developed under the NASA program that apply to all three of the aforementioned committees. It discusses the similarities and differences in the problem spaces under consideration in each committee, and considers the application of a common set of CNS alternatives that can be widely applied. As the works of these committees progress, it is clear that the overlap will need to be addressed to ensure a consistent and safe framework for worldwide aviation. In this study, we discuss similarities and differences in the various operational models and show how the CNS architectures developed under the NASA program apply

Benchmarking Deep Learning Models for Tooth Structure Segmentation.

A wide range of deep learning (DL) architectures with varying depths are available, with developers usually choosing one or a few of them for their specific task in a nonsystematic way. Benchmarking (i.e., the systematic comparison of state-of-the art architectures on a specific task) may provide guidance in the model development process and may allow developers to make better decisions. However, comprehensive benchmarking has not been performed in dentistry yet. We aimed to benchmark a range of architecture designs for 1 specific, exemplary case: tooth structure segmentation on dental bitewing radiographs. We built 72 models for tooth structure (enamel, dentin, pulp, fillings, crowns) segmentation by combining 6 different DL network architectures (U-Net, U-Net++, Feature Pyramid Networks, LinkNet, Pyramid Scene Parsing Network, Mask Attention Network) with 12 encoders from 3 different encoder families (ResNet, VGG, DenseNet) of varying depth (e.g., VGG13, VGG16, VGG19). On each model design, 3 initialization strategies (ImageNet, CheXpert, random initialization) were applied, resulting overall into 216 trained models, which were trained up to 200 epochs with the Adam optimizer (learning rate = 0.0001) and a batch size of 32. Our data set consisted of 1,625 human-annotated dental bitewing radiographs. We used a 5-fold cross-validation scheme and quantified model performances primarily by the F1-score. Initialization with ImageNet or CheXpert weights significantly outperformed random initialization (P < 0.05). Deeper and more complex models did not necessarily perform better than less complex alternatives. VGG-based models were more robust across model configurations, while more complex models (e.g., from the ResNet family) achieved peak performances. In conclusion, initializing models with pretrained weights may be recommended when training models for dental radiographic analysis. Less complex model architectures may be competitive alternatives if computational resources and training time are restricting factors. Models developed and found superior on nondental data sets may not show this behavior for dental domain-specific tasks

ARTMAP Neural Network Classification of Land Use Change

The ability to detect and monitor changes in land use is essential for assessment of the sustainability of development. In the next decade, NASA will gather high-resolution multi-spectral and multi-temporal data, which could be used for detecting and monitoring long-term changes. Existing methods are insufficient for detecting subtle long-term changes from high-dimensional data. This project employs neural network architectures as alternatives to conventional systems for classifying changes in the status of agricultural lands from a sequence of satellite images. Landsat TM imagery of the Nile River delta provides a testbed for these land use change classification methods. A sequence often images was taken, at various times of year, from 1984 to 1993. Field data were collected during the summer of 1993 at88 sites in the Nile Delta and surrounding desert areas. Ground truth data for 231 additional sites were determined by expert site assessment at the Boston University Center for Remote Sensing. The field observations are grouped into classes including urban, reduced productivity agriculture, agriculture in delta, desert/coast reclamation, wetland reclamation, and agriculture in desert/coast. Reclamation classes represent land use changes. A particular challenge posed by this database is the unequal representation of various land use categories: urban and agriculture in delta pixels comprise the vast majority of the ground truth data available in the database. A new, two-step training data selection method was introduced to enable unbiased training of neural network systems on sites with unequal numbers of pixels. Data were successfully classified by using multi-date feature vectors containing data from all of the available satellite images as inputs to the neural network system.National Science Foundation Graduate Fellowship; National Science Foundation (SBR 95-13889); Office of Naval Research (N00014-95-I-409, N00014-95-0657); Air Force Office of Scientific Research (F49620-0l-1-0397)

Multitask Learning Deep Neural Networks to Combine Revealed and Stated Preference Data

It is an enduring question how to combine revealed preference (RP) and stated preference (SP) data to analyze travel behavior. This study presents a framework of multitask learning deep neural networks (MTLDNNs) for this question, and demonstrates that MTLDNNs are more generic than the traditional nested logit (NL) method, due to its capacity of automatic feature learning and soft constraints. About 1,500 MTLDNN models are designed and applied to the survey data that was collected in Singapore and focused on the RP of four current travel modes and the SP with autonomous vehicles (AV) as the one new travel mode in addition to those in RP. We found that MTLDNNs consistently outperform six benchmark models and particularly the classical NL models by about 5% prediction accuracy in both RP and SP datasets. This performance improvement can be mainly attributed to the soft constraints specific to MTLDNNs, including its innovative architectural design and regularization methods, but not much to the generic capacity of automatic feature learning endowed by a standard feedforward DNN architecture. Besides prediction, MTLDNNs are also interpretable. The empirical results show that AV is mainly the substitute of driving and AV alternative-specific variables are more important than the socio-economic variables in determining AV adoption. Overall, this study introduces a new MTLDNN framework to combine RP and SP, and demonstrates its theoretical flexibility and empirical power for prediction and interpretation. Future studies can design new MTLDNN architectures to reflect the speciality of RP and SP and extend this work to other behavioral analysis