19 research outputs found

    Identifying atmospheric rivers and their poleward latent heat transport with generalizable neural networks: ARCNNv1

    Get PDF
    Atmospheric rivers (ARs) are extreme weather events that can alleviate drought or cause billions of US dollars in flood damage. By transporting significant amounts of latent energy towards the poles, they are crucial to maintaining the climate system's energy balance. Since there is no first-principle definition of an AR grounded in geophysical fluid mechanics, AR identification is currently performed by a multitude of expert-defined, threshold-based algorithms. The variety of AR detection algorithms has introduced uncertainty into the study of ARs, and the thresholds of the algorithms may not generalize to new climate datasets and resolutions. We train convolutional neural networks (CNNs) to detect ARs while representing this uncertainty; we name these models ARCNNs. To detect ARs without requiring new labeled data and labor-intensive AR detection campaigns, we present a semi-supervised learning framework based on image style transfer. This framework generalizes ARCNNs across climate datasets and input fields. Using idealized and realistic numerical models, together with observations, we assess the performance of the ARCNNs. We test the ARCNNs in an idealized simulation of a shallow-water fluid in which nearly all the tracer transport can be attributed to AR-like filamentary structures. In reanalysis and a high-resolution climate model, we use ARCNNs to calculate the contribution of ARs to meridional latent heat transport, and we demonstrate that this quantity varies considerably due to AR detection uncertainty.</p

    Author Correction: The FLUXNET2015 dataset and the ONEFlux processing pipeline for eddy covariance data

    Full text link
    The following authors were omitted from the original version of this Data Descriptor: Markus Reichstein and Nicolas Vuichard. Both contributed to the code development and N. Vuichard contributed to the processing of the ERA-Interim data downscaling. Furthermore, the contribution of the co-author Frank Tiedemann was re-evaluated relative to the colleague Corinna Rebmann, both working at the same sites, and based on this re-evaluation a substitution in the co-author list is implemented (with Rebmann replacing Tiedemann). Finally, two affiliations were listed incorrectly and are corrected here (entries 190 and 193). The author list and affiliations have been amended to address these omissions in both the HTML and PDF versions

    The FLUXNET2015 dataset and the ONEFlux processing pipeline for eddy covariance data.

    Full text link
    The FLUXNET2015 dataset provides ecosystem-scale data on CO2, water, and energy exchange between the biosphere and the atmosphere, and other meteorological and biological measurements, from 212 sites around the globe (over 1500 site-years, up to and including year 2014). These sites, independently managed and operated, voluntarily contributed their data to create global datasets. Data were quality controlled and processed using uniform methods, to improve consistency and intercomparability across sites. The dataset is already being used in a number of applications, including ecophysiology studies, remote sensing studies, and development of ecosystem and Earth system models. FLUXNET2015 includes derived-data products, such as gap-filled time series, ecosystem respiration and photosynthetic uptake estimates, estimation of uncertainties, and metadata about the measurements, presented for the first time in this paper. In addition, 206 of these sites are for the first time distributed under a Creative Commons (CC-BY 4.0) license. This paper details this enhanced dataset and the processing methods, now made available as open-source codes, making the dataset more accessible, transparent, and reproducible

    Author Correction: The FLUXNET2015 dataset and the ONEFlux processing pipeline for eddy covariance data

    Get PDF

    The FLUXNET2015 dataset and the ONEFlux processing pipeline for eddy covariance data

    Get PDF
    The FLUXNET2015 dataset provides ecosystem-scale data on CO2, water, and energy exchange between the biosphere and the atmosphere, and other meteorological and biological measurements, from 212 sites around the globe (over 1500 site-years, up to and including year 2014). These sites, independently managed and operated, voluntarily contributed their data to create global datasets. Data were quality controlled and processed using uniform methods, to improve consistency and intercomparability across sites. The dataset is already being used in a number of applications, including ecophysiology studies, remote sensing studies, and development of ecosystem and Earth system models. FLUXNET2015 includes derived-data products, such as gap-filled time series, ecosystem respiration and photosynthetic uptake estimates, estimation of uncertainties, and metadata about the measurements, presented for the first time in this paper. In addition, 206 of these sites are for the first time distributed under a Creative Commons (CC-BY 4.0) license. This paper details this enhanced dataset and the processing methods, now made available as open-source codes, making the dataset more accessible, transparent, and reproducible.Peer reviewe

    LBL-EESA/TECA: TECA 6.0.0

    No full text
    &lt;h2&gt;6.0.0&lt;/h2&gt; &lt;p&gt;This is a majopr release that contains numerous improvements and fixes. BARD is fully GPUized. Temporal reductions have been ported to C++ and optimized. Data and execution modelss have been extended for multiple steps per request and spatial partitioning. Numerous I/O optimizations ahave been introduced including the use of collective buffering for spatial parallel execution.&lt;/p&gt; &lt;h4&gt;Execution Model&lt;/h4&gt; &lt;p&gt;e134264b add spatial executive c6f9bc62 cf_writer add partitioning contraints 4d53de68 add space_time_executive 97efc350 add cf_space_time_time_step_mappper 98bbb97b adds cf_spatial_time_step_mapper 3d915ee9 cf_space_time_time_step_mapper add partitioning contraints a8fa4d8d cf_spatial_time_step_mapper add partitioning contraints 19f5e229 coordinate_util partition add contraints 4d2a8f1c index_reduce execution controls c765bd51 cf_writer command line parsing of spatial parallel properties 7c0c8a32 spatial_executive constrain partitioning 792b7f94 space_time_executive constrain partitioning f572c81e metadata_probe report number of intervals b295896e mesh wrap temporal bounds and extent daa684d7 index_request_key update 25bd3b44 index_executive clean up verbose report a61ec638 test cf_reader temporal extent handling 6e3323dc dataset_diff handle temporal extents d5dad5eb test temporal reduction spatial parallelism 019dc836 cf_writer spatial parallelism f3c14a09 cf_layout_manager spatial parallelism ba50dd85 cf_time_step_mapper layout manager API 9aa17f18 interval time step mapper refactor e3a25a8c block time step mapper refactor 1cfcc08f coordinate util spatial partitioning e341f638 cf_reader reads temporal extents 423a8da9 data model updates for multiple time steps per mesh&lt;/p&gt; &lt;h4&gt;Data Model&lt;/h4&gt; &lt;p&gt;03939e15 add and apply simplified dispatch macros 69e88df9 hamr update to latest 422f3835 hamr fully asynchronous by default 2cd9c8e7 hamr enforce const for read only data access 95de5935 hamr update to latest master 2927a95e HAMR update to latest master adf56038 variant_array_util add host synchronization helper 69760845 variant_array add synchronization method c7b1b2d9 add teca_variant_array_util 29897a4c variant_array better dispatch 1ce73a70 variant_array better dispatch a70cdfe8 variant_array make test for accessibility virtual 1422ea30 variant_array provide direct access to internal memory 59111349 variant_array python construct from numpy scalar c3562a76 cartesian_mesh fall back to mesh extents 03143cb2 cartesian_mesh_source spatial parallelism b8615ed7 cartesian_mesh_regrid per array dimensions ca4fcbb3 cartesian_mesh per array extent and shape const 42446f27 cartesian_mesh_source generate data on the assigned GPU d3082de4 cartesian_mesh_source include bounds metadata in output mesh acf3fe2e cartesian_mesh overload array shape to return a tuple 6a9f3ac0 cartesian_mesh_regrid pass array attributes from the source e5e8e4a7 cartesian_mesh array extent time dim and add shape 73b58ebb cartesian_mesh fix Python bindings for array shape/extent 86ef5616 cartesian_mesh_source fix calendaring metadata in output&lt;/p&gt; &lt;h4&gt;New Algorithms&lt;/h4&gt; &lt;p&gt;f730aa81 add teca_surface_integral alg f79c2c8d add teca_regional_moisture_flux dc66e328 add teca_table_join f2af4c41 add spectral filter e439275e add teca_vtk_util::partition_writer to help debug space-time paritioning 0fe459e0 add temporal_percentile temporal reduction 140008c5 wrote temporal_index_select and tests&lt;/p&gt; &lt;h4&gt;New Applications&lt;/h4&gt; &lt;p&gt;acfcaffe add regional_moisture_flux app cfd6ce85 Add the spectral filter app&lt;/p&gt; &lt;h4&gt;GPUization&lt;/h4&gt; &lt;p&gt;a64839b6 bayesian_ar_detect add CUDA implementation cf74102e 2d_component_area thrust use stream per thread stream 42d16f76 2d_component_area set cuda device before doing any work e54e33b4 component_area_filter set cuda device before doing any work c3efa90d connected_components set cuda device before doing any work 45a87f1d bayeseian_ar_detect set cuda device before doing any work 3791b67d latitude_damper set cuda device before doing any work 8993ed66 unpack_data set cuda device before doing any work 640ee577 index_executive explicitly assign device ids 79445b3b binary_segmentation use streams for sorting and data movement 23347358 cuda_util add a 1d domain decomposition 9644b346 latitude_damper add CUDA implementation a2432065 component_area_filter add CUDA implementation 5a2f6603 2d_component_area use restrict on kernels ad65931f 2d_component_area GPU-ize the area calculation 96c59666 cf_reader don't use page locked memory for cuda 7549e888 cuda_util simplify device assignment 1b14777c connected_components use 8 connetivity 52be3623 ha4 test code use 8 connectivity 2f4047f9 index_executive environment variable override CUDA device assignment 0919c784 connected_components inetgrate CUDA ha4 implementation 77884268 shape_file_mask add CUDA implementation c44aded2 cuda_util implement a container for cuda streams edf6c588 geometry_util GPUize point in poly 693a7b2c thread_util threads per device behavior ac2f59fe cuda warning cleanup 3f2ba7f7 spatial_executive load balance across GPUs 5c082594 space_time_executive load balance across GPUs&lt;/p&gt; &lt;h4&gt;Threading&lt;/h4&gt; &lt;p&gt;62410659 bayesian_ar_detect fix thread safety issues fa1c2099 thread_util warn about too few threads wo MPI 1d5f4158 thread_util clamp the number of threads c9704448 thread_util report num threads when not binding af1592a4 threaded_algorithm propagate_device_assignment 81d4e2d0 threaded_algorithm expose ranks_per_device in API&lt;/p&gt; &lt;h4&gt;Optimizations&lt;/h4&gt; &lt;p&gt;60c9e718 cf_restripe app add collective buffer mode 3dbc0e22 Added C++ version of the temporal reduction algorithm and application 9735209c cf_reader open file in collective mode 5558ff66 spectral_filter app command line options for collective buffering c0efea8f cf/multi_cf_reader option to use collective buffering f304f275 cf_writer use collective buffering&lt;/p&gt; &lt;h4&gt;Documentation&lt;/h4&gt; &lt;p&gt;d5eb0fcc cf_reader fix copy paste error in documentation e5306fac component_area_filter fix indent add comments 30adda58 algorithm fix a documentation typo bb730837 shape_file_mask improve documentation d8fcade0 table_reduce improve documentation b166667f integrated_water_vapor improve documentation ef2cd480 integrated_vapor_transport improve documentation f3623803 threaded_algorithm improve documentation e5a26ff2 doc doxygen style comments for programmable_algorithm dc367728 doc doxygen style comments for teca_table de5e8d68 doc data access developer tutorial 1d25525b interval_iterator subclasses fix units doxygen doc strings dd5f1fee doc update temporal_reduction user guide c71e9057 cf_writer fix typo in docs 53effc02 doc update m1517 install locations for perlmutter 1b71d8eb coordinate_util improve documentation ff383a0f rtd add section explaining execution model ae237bd9 rtd docs fix doxygen install location c51132bf rtd pin sphinx version as latest is incompatible with rtddocs 5ea6e10c rtd doc array access tutorial spell check af9d2e6c doc rtd improve array access tutorial b528ec9d rtd fix a rst warning 9a6e888b rtd updates to the install for mac os 1a7dc382 doc rtd exclude variant_array_oeprator from doxygen&lt;/p&gt; &lt;h4&gt;Testing&lt;/h4&gt; &lt;p&gt;bf97e954 test disable periodic bc in bard app test 238db9f6 test bayesian ar detect sort by label area 49e83a90 deeplab_ar_detect remove tests b7d14f17 testing update linux distributions c38337f9 testing cleanup use of %e% in tests d40d800a temporal_reduction: added tests 80a01599 test add test for cpp_temporal_reduciton w. io 3b277b3a test temporal reduction steps_per_request command line argument 9e614ea1 add test_temporal_reduciton 3b338bf9 ha4 test code update ctests command 5dd84cb9 connected_components test ignore component labels 6569a79f ha4 test code improvements a1012ed6 ha4 test code handles periodic BC in x-direction a380f62c ha4 test code works on images not divisible by 32 e6216c3b add ha4 connected component label test code a769ff73 test_streaming_reduce_threads: specifying netcdf file name to avoid conflict with temporal reduction all iterator test 6e02fa62 test temporal_reduction app python and C++ d79206a4 testing temporal_reduction tests specify number of threads 709f6853 temporal_reduction C++ impl improvements and regression test 5120006d update the DOI badge to point to the latest release 18533f8c Changed teca data revision from 149 to 151&lt;/p&gt; &lt;h4&gt;General Improvement&lt;/h4&gt; &lt;p&gt;24142094 bayesian_ar_detect_parameters add properties to select specific start row be087dc5 bayesian_ar_detect instrument the BARD app 37f4237e bayesian_ar_detect app control writer thread pool size 176c1f6b connected_components cleanup a warning 10eaf195 connected_components minor improvements ee8cbf23 temporal_reduction: set steps_per_request in python app; included definition in cpp app 27f3ef3e temporal_reduction: standardized n_threads command line b371bea9 temporal_reduction construct output at end and others 494a3b42 temporal_reduction: caching the intermediate result 07a119ae temporal_reduction: any number of time steps per request is allowed bd321844 descriptive_statistics remove debuging code 18768fd8 index_executive fix a compiler warning ff551dce cpp_temporal_reduction algorithm errors are fatal 95bd6a88 temporal_reduction: set_thread_pool_size [cf_writer] changed from -1 to 1 to fix intermittent bugs 7953cbbb temporal_reduction: change the 1 time step per request to a run time specified number of steps 1bab4257 dataset_diff ignore specified arrays 03fc0bc7 table_sort sort either ascending or descending b29c4fd7 coordinate_util wrap bounds to extent overload d0ac7a98 integrated_vapor_transport handle ascending coordinates in the first order method b593e57d integrated_vapor_transport app enable automatic z-axis unit conversion 78675ec9 integrated_vapor_transport warn if vertical axis units are incorrect 63087d20 normalize_coordinates check z-axis units df9378e0 integrated_vapor_transport layer thickness eb4853a7 evaluate_expression netcdf attributes for the cf_writer 4cccc26f table include dataset property for array attributes 98e0a891 table_join pass array attributes for NetCDF I/O 3b815827 integrated_water_vapor reformat units string f6eabe0f algorithm add a single value setter for vector properties 657ba214 index_reduce use std::vector instead of std::array abac3f23 indexed_dataset_cache override request index eb86345c integrated_vapor_transport change format of units b0a4390c dataset_source report variables from tables and meshes 31b748a2 dataset_source move to alg to access typed datasets 290db6bf coordinate_util improvements 20c5ad67 table_reduce report and request use default implementations b878f89f program_options support std::array in algorithm properties 06b52b2f shape_file_mask improvements 9a506d7f dataset typed accessors 40ea89bd derived quantity improvements ebb12862 array_attributes include mesh_dim_active 254f9e7f temporal_reduction app/alg cpp/python catch user errors a1eb0f0e cf_writer improve error message 589f70c6 cf_writer improve collective buffering error message d01da07f cf_restripe app runs in CPU only mode by default 19334ed6 cf_reader improve collective buffering error c4de2426 spectral_filter per-rank timing output in verbose mode 51a26703 spectral_filter add ideal butterworth frequency response dadb4911 spectral_filter fixes issue found when processing real data 2a8816ff spectral_filter refactor regression tests 976482d6 spectral filter fix high pass kernel generation 57e3f31d teca_temporal_reduction: added all iterator average test 9924d676 teca_temporal_reduction: added all iterator fbb866ea teca_calendar_util: added new class all_iterator c6704cdc temporal_reduction: added flag to spatial parallelism 4b0d251d teca_calendar_util: added the new class n_steps_iterator ec98d675 added index selection to the temporal reduction 6f1ae9da metadata add support for std::array 6e99362f vorticity better identitiers in dispatch macro 4407b536 cuda_util remove redundant error check d11f4803 valid_value_mask export mask type 296ec4a0 temporal_reduction app command line option controlling threadpool size 48e32130 temporal_reduction: rename the C++ implementation bd6718ac temporal_reduction: handle the case where the number of inputs &lt; 2. 19ead29e temporal_reduction: renamed the original python implementation fbd22354 temporal_reduction: resolving a warning 9ba6d353 temporal_reduction clean up warnings with nvcc 3bcfcf43 tenporal_reduction app integrate multi threading ec71e980 Renamed python version of temporal reduction; python bindings cd28e3e8 teca_threaded_programmable_algorithm: increased the size of the class_name variable from 64 to 96 ccfdba31 potential_intensity user provided masking threshold e7c53c0c potential_intensity units checks and conversions c674b798 potential_intensity app reduce verbosity 8f6a1ed3 teca_potential_intensity clean up runtime warnings df50b49a python functions returing typed scalars 11814513 potential_intensity app use spatial partitioning e18f4d49 potential_intensity app land mask from mcf file 839ef1c3 app_util error out with positional options&lt;/p&gt; &lt;h4&gt;Bug fixes&lt;/h4&gt; &lt;p&gt;6cb6cccf component_area_filter fix indentation ee9a4d84 connected_components fix 8-way connectivity accross periodic boundary e85aa72f system_interface fix double free in stack trace generation d19e7270 testing fix the component are filter test 10e94597 temporal_reduction: fix data access d6b22e63 teca_profiler: fixed convertion of hexstring to int 220587ad cpu_thread_pool fix bind argument position bf99eff3 cpu/cuda_thread_pool fix streaming bug 3d5d4db2 cf_writer fix let threaded_algorithm process command line 80820ef1 threaded_algorithm fix set algorithm props from command line 59c42b53 threaded_algorithm fix threads_per_device parameter name bada5a60 cpp_temporal_reduciton fix thread safety issues 7de9224c cpp_temporal_reduction fix a typo in documentation 59eb4c79 ha4 test code fix race condition 2eaa71b6 connected_components fix race condition ddaf758f connected_components fix compile w/o cuda 4c8032c1 connected_components 8-way connectivity bug fixes 55b0908f ha4 test code 8-way connectivity bug fixes 18e0c92d rename_variables fix set variables in the output attributes e7396820 fixes for cuda 12 and warning cleanup for gcc 12 9d13cd42 temporal_reduction fix missing virtual destructor in base class 76ab59d8 array_collection fix double move f462ac83 normalize_coordinates fix a bug in the output extents e8dcfca3 tests fix regex that picks up new file e3dc08f3 cpp_temporal_reduction cleanup, fixes, and improvements 00ba2421 temporal_reduction: included flag to choose python or c++ implementation; fixing the n_steps interval bc43a364 temporal_reduction: rename the python implementation; fixing name of two python tests 244f58e5 temporal_reduction: fixing the parameter order in a test 79b36732 temporal_reduction: added a new finalize function to fix a bug 942aa111 temporal_reduction fix a warning and set strream size 7120ecb8 cpu/cuda_thread_pool fix thread safety issues d2519402 threaded_algorithm fix indentation 26fba6d7 potential_intensity units checks and conversion fix 45dbd2b9 Fixed n_steps_iterator class of python version of temporal_reduction 86e6ea74 calendaring fix buffer overflow warnings 5ffc2d4e Fixing issue 98f04ee3 temporal_percentile fixes&lt;/p&gt; &lt;h4&gt;Python&lt;/h4&gt; &lt;p&gt;42ca1d80 python support wrapping API with fixed length C-arrays 61e9f34a remove numpy deprecated types&lt;/p&gt; &lt;h4&gt;Build System&lt;/h4&gt; &lt;p&gt;a76c7cf9 build cleanup cmake code 8f965035 added CMAKE_INSTALL_RPATH to CMakeLists.txt 3e43838f build define NDEBUG in CUDA release build 08b95f05 build always update the version descriptor 944a3f25 build system don't relink unless neccessary&lt;/p&gt
    corecore