Long-Range Correlation Underlying Childhood Language and Generative Models

Long-range correlation, a property of time series exhibiting long-term memory, is mainly studied in the statistical physics domain and has been reported to exist in natural language. Using a state-of-the-art method for such analysis, long-range correlation is first shown to occur in long CHILDES data sets. To understand why, Bayesian generative models of language, originally proposed in the cognitive scientific domain, are investigated. Among representative models, the Simon model was found to exhibit surprisingly good long-range correlation, but not the Pitman-Yor model. Since the Simon model is known not to correctly reflect the vocabulary growth of natural language, a simple new model is devised as a conjunct of the Simon and Pitman-Yor models, such that long-range correlation holds with a correct vocabulary growth rate. The investigation overall suggests that uniform sampling is one cause of long-range correlation and could thus have a relation with actual linguistic processes

Do Neural Nets Learn Statistical Laws behind Natural Language?

The performance of deep learning in natural language processing has been spectacular, but the reasons for this success remain unclear because of the inherent complexity of deep learning. This paper provides empirical evidence of its effectiveness and of a limitation of neural networks for language engineering. Precisely, we demonstrate that a neural language model based on long short-term memory (LSTM) effectively reproduces Zipf's law and Heaps' law, two representative statistical properties underlying natural language. We discuss the quality of reproducibility and the emergence of Zipf's law and Heaps' law as training progresses. We also point out that the neural language model has a limitation in reproducing long-range correlation, another statistical property of natural language. This understanding could provide a direction for improving the architectures of neural networks.Comment: 21 pages, 11 figure

Nanolevel Surface Processing of Fine Particles by Waterjet Cavitation and Multifunction Cavitation to Improve the Photocatalytic Properties of Titanium Oxide

Titanium oxide particles were treated by water jet cavitation (WJC) generated and multifunction cavitation (MFC) using an ejector nozzle. Generation, growth, and collapse of cavitation are repeated with the particles of titanium oxide and platinum. Because the cavitation has an extremely high collapse pressure, the surface of the titanium oxide particles is processed by the microjets of cavitation in a reactor comprising the ejector nozzle. In the multifunction cavitation, ultrasonic irradiation of a waterjet during floating cavitation was used to generate microjets with hot spots. Hot working can be performed at the nanoscale on a material surface using this MFC process, resulting in morphological changes and variations in the surface electrochemical characteristics. The fundamental characteristics of multifunction cavitation were investigated theoretically and experimentally. Furthermore, the additional nozzle was put on the ejector nozzle in order to increase the temperature and pressure of bubble and the mechanism was clarified. The quantities of hydrogen and oxygen generated from titanium dioxide particles treated by multifunction cavitation in response to UV and visible light irradiation were remarkably increased compared to the amounts produced by particles treated by WJC processing. In this chapter, the methods and their results of processing particles by cavitation are introduced

Co-Training Realized Volatility Prediction Model with Neural Distributional Transformation

This paper shows a novel machine learning model for realized volatility (RV) prediction using a normalizing flow, an invertible neural network. Since RV is known to be skewed and have a fat tail, previous methods transform RV into values that follow a latent distribution with an explicit shape and then apply a prediction model. However, knowing that shape is non-trivial, and the transformation result influences the prediction model. This paper proposes to jointly train the transformation and the prediction model. The training process follows a maximum-likelihood objective function that is derived from the assumption that the prediction residuals on the transformed RV time series are homogeneously Gaussian. The objective function is further approximated using an expectation-maximum algorithm. On a dataset of 100 stocks, our method significantly outperforms other methods using analytical or naive neural-network transformations.Comment: Accepted at ICAIF'2