25,048 research outputs found
DAMEWARE - Data Mining & Exploration Web Application Resource
Astronomy is undergoing through a methodological revolution triggered by an
unprecedented wealth of complex and accurate data. DAMEWARE (DAta Mining &
Exploration Web Application and REsource) is a general purpose, Web-based,
Virtual Observatory compliant, distributed data mining framework specialized in
massive data sets exploration with machine learning methods. We present the
DAMEWARE (DAta Mining & Exploration Web Application REsource) which allows the
scientific community to perform data mining and exploratory experiments on
massive data sets, by using a simple web browser. DAMEWARE offers several tools
which can be seen as working environments where to choose data analysis
functionalities such as clustering, classification, regression, feature
extraction etc., together with models and algorithms.Comment: User Manual of the DAMEWARE Web Application, 51 page
Data Mining and Machine Learning in Astronomy
We review the current state of data mining and machine learning in astronomy.
'Data Mining' can have a somewhat mixed connotation from the point of view of a
researcher in this field. If used correctly, it can be a powerful approach,
holding the potential to fully exploit the exponentially increasing amount of
available data, promising great scientific advance. However, if misused, it can
be little more than the black-box application of complex computing algorithms
that may give little physical insight, and provide questionable results. Here,
we give an overview of the entire data mining process, from data collection
through to the interpretation of results. We cover common machine learning
algorithms, such as artificial neural networks and support vector machines,
applications from a broad range of astronomy, emphasizing those where data
mining techniques directly resulted in improved science, and important current
and future directions, including probability density functions, parallel
algorithms, petascale computing, and the time domain. We conclude that, so long
as one carefully selects an appropriate algorithm, and is guided by the
astronomical problem at hand, data mining can be very much the powerful tool,
and not the questionable black box.Comment: Published in IJMPD. 61 pages, uses ws-ijmpd.cls. Several extra
figures, some minor additions to the tex
A database with enterprise application for mining astronomical data obtained by MOA : a thesis submitted in partial fulfilment of the requirements for the degree of the Master of Information Science in Computer Science, Massey University at Albany, Auckland, New Zealand
The MOA (Microlensing Observations in Astrophysics) Project is one of a new generation of modern astronomy endeavours that generates huge volumes of data. These have enormous scientific data mining potential. However, it is common for astronomers to deal with millions and even billions of records. The challenge of how to manage these large data sets is an important case for researchers. A good database management system is vital for the research. With the modern observation equipments used, MOA suffers from the growing volume of the data and a database management solution is needed. This study analyzed the modern technology for database and enterprise application. After analysing the data mining requirements of MOA, a prototype data management system based on MVC pattern was developed. Furthermore, the application supports sharing MOA findings and scientific data on the Internet. It was tested on a 7GB subset of achieved MOA data set. After testing, it was found that the application could query data in an efficient time and support data mining
Astroinformatics, data mining and the future of astronomical research
Astronomy, as many other scientific disciplines, is facing a true data deluge
which is bound to change both the praxis and the methodology of every day
research work. The emerging field of astroinformatics, while on the one end
appears crucial to face the technological challenges, on the other is opening
new exciting perspectives for new astronomical discoveries through the
implementation of advanced data mining procedures. The complexity of
astronomical data and the variety of scientific problems, however, call for
innovative algorithms and methods as well as for an extreme usage of ICT
technologies.Comment: To appear in the Proceedings of the 2-nd International Conference on
Frontiers on diagnostic technologie
Mining Knowledge in Astrophysical Massive Data Sets
Modern scientific data mainly consist of huge datasets gathered by a very
large number of techniques and stored in very diversified and often
incompatible data repositories. More in general, in the e-science environment,
it is considered as a critical and urgent requirement to integrate services
across distributed, heterogeneous, dynamic "virtual organizations" formed by
different resources within a single enterprise. In the last decade, Astronomy
has become an immensely data rich field due to the evolution of detectors
(plates to digital to mosaics), telescopes and space instruments. The Virtual
Observatory approach consists into the federation under common standards of all
astronomical archives available worldwide, as well as data analysis, data
mining and data exploration applications. The main drive behind such effort
being that once the infrastructure will be completed, it will allow a new type
of multi-wavelength, multi-epoch science which can only be barely imagined.
Data Mining, or Knowledge Discovery in Databases, while being the main
methodology to extract the scientific information contained in such MDS
(Massive Data Sets), poses crucial problems since it has to orchestrate complex
problems posed by transparent access to different computing environments,
scalability of algorithms, reusability of resources, etc. In the present paper
we summarize the present status of the MDS in the Virtual Observatory and what
is currently done and planned to bring advanced Data Mining methodologies in
the case of the DAME (DAta Mining & Exploration) project.Comment: Pages 845-849 1rs International Conference on Frontiers in
Diagnostics Technologie
DAMEWARE - Data Mining & Exploration Web Application Resource
Astronomy is undergoing through a methodological revolution triggered by an unprecedented wealth of complex and accurate data. DAMEWARE (DAta Mining & Exploration Web Application and REsource) is a general purpose, Web-based, Virtual Observatory compliant, distributed data mining framework specialized in massive data sets exploration with machine learning methods. We present the DAMEWARE (DAta Mining & Exploration Web Application REsource) which allows the scientific community to perform data mining and exploratory experiments on massive data sets, by using a simple web browser. DAMEWARE offers several tools which can be seen as working environments where to choose data analysis functionalities such as clustering, classification, regression, feature extraction etc., together with models and algorithms
On the application of machine learning in astronomy and astrophysics: A text-mining-based scientometric analysis
Since the beginning of the 21st century, the fields of astronomy and astrophysics
have experienced significant growth at observational and computational levels,
leading to the acquisition of increasingly huge volumes of data. In order to process
this vast quantity of information, artificial intelligence (AI) techniques are being
combined with data mining to detect patterns with the aim of modeling, classifying
or predicting the behavior of certain astronomical phenomena or objects.
Parallel to the exponential development of the aforementioned techniques, the
scientific output related to the application of AI and machine learning (ML) in
astronomy and astrophysics has also experienced considerable growth in recent
years. Therefore, the increasingly abundant articles make it difficult to monitor
this field in terms of which research topics are the most prolific or novel, or which
countries or authors are leading them. In this article, a text-mining-based
scientometric analysis of scientific documents published over the last three
decades on the application of AI and ML in the fields of astronomy and astrophysics
is presented. The VOSviewer software and data from the Web of Science
(WoS) are used to elucidate the evolution of publications in this research field,
their distribution by country (including co-authorship), the most relevant topics
addressed, and the most cited elements and most significant co-citations according
to publication source and authorship. The obtained results demonstrate how
application of AI/ML to the fields of astronomy/astrophysics represents an
established and rapidly growing field of research that is crucial to obtaining
scientific understanding of the universe
Exploration of Large Digital Sky Surveys
We review some of the scientific opportunities and technical challenges posed
by the exploration of the large digital sky surveys, in the context of a
Virtual Observatory (VO). The VO paradigm will profoundly change the way
observational astronomy is done. Clustering analysis techniques can be used to
discover samples of rare, unusual, or even previously unknown types of
astronomical objects and phenomena. Exploration of the previously poorly probed
portions of the observable parameter space are especially promising. We
illustrate some of the possible types of studies with examples drawn from
DPOSS; much more complex and interesting applications are forthcoming.
Development of the new tools needed for an efficient exploration of these vast
data sets requires a synergy between astronomy and information sciences, with
great potential returns for both fields.Comment: To appear in: Mining the Sky, eds. A. Banday et al., ESO Astrophysics
Symposia, Berlin: Springer Verlag, in press (2001). Latex file, 18 pages, 6
encapsulated postscript figures, style files include
Virtual Astronomy, Information Technology, and the New Scientific Methodology
All sciences, including astronomy, are now entering the era of information abundance. The exponentially increasing volume and complexity of modern data sets promises to transform the scientific practice, but also poses a number of common technological challenges. The Virtual Observatory concept is the astronomical community's response to these challenges: it aims to harness the progress in information technology in the service of astronomy, and at the same time provide a valuable testbed for information technology and applied computer science. Challenges broadly fall into two categories: data handling (or "data farming"), including issues such as archives, intelligent storage, databases, interoperability, fast networks, etc., and data mining, data understanding, and knowledge discovery, which include issues such as automated clustering and classification, multivariate correlation searches, pattern recognition, visualization in highly hyperdimensional parameter spaces, etc., as well as various applications of machine learning in these contexts. Such techniques are forming a methodological foundation for science with massive and complex data sets in general, and are likely to have a much broather impact on the modern society, commerce, information economy, security, etc. There is a powerful emerging synergy between the
computationally enabled science and the science-driven computing, which will drive the progress in science, scholarship, and many other venues in the 21st century
- …